/ โ† AEO Fixes / Optimise for Gemini
In this guide
  1. Three assistants, three routes to your site
  2. Gemini: competing inside Google Search
  3. Copilot: the Bing side
  4. Claude: Anthropic's crawlers and fetches
  5. The fix, step by step
  6. Work that helps all three at once
  7. What not to spend time on
  8. Measuring progress for each assistant
  9. Questions and answers

Three assistants, three routes to your site

All three assistants write answers from a mix of what their model learned in training and what they fetch from the web at the time of the question. The part you can influence quickly is the fetching, and that is where they differ.

AssistantWhere live sources come fromCrawler names to knowYour main lever
Gemini, AI Overviews, AI ModeThe Google Search index.Googlebot; Google-Extended is a robots.txt token, not a crawler.Google Search Console and how well your pages answer the question.
Microsoft CopilotThe Bing index.bingbotBing Webmaster Tools, IndexNow and Bing's own ranking.
ClaudeAnthropic's web search, plus pages fetched when a user asks.Claude-SearchBot, Claude-User, ClaudeBot (training).Crawler access and pages that read well when fetched directly.

The overlap is larger than the table suggests. All three reward the same thing once they have your page: a clear, specific answer near the top, written so a sentence or two can be lifted out and still make sense. The differences are mostly about getting found in the first place.

๐Ÿ’ก If you are missing from Gemini altogether rather than simply cited less than you would like, start with fixing invisibility in Gemini, which covers indexing, snippet rules and Google-Extended in detail. This page assumes the basics are working.

Gemini: competing inside Google Search

Google's own guidance on AI features says there are no extra technical requirements for appearing in AI Overviews or AI Mode beyond being indexed and eligible to show a snippet, and that normal Search best practice applies. That is useful to know, because it rules out a lot of paid advice about special AI tags. What it leaves is the quality and coverage of your content.

Fan-out questions

Google has described AI Mode as breaking a question into several related searches, running them together and combining what it finds. A person asking "Is it worth fitting a heat pump in a 1930s semi?" may trigger searches about running costs, insulation, grants, noise and installation time. A page that answers only the headline question competes for one of those searches. A page, or a small cluster of linked pages, that answers the follow-ups too has more chances to be used.

You can find likely follow-ups in the "People also ask" boxes for your main question, in Search Console queries that already bring impressions to the page, and in the questions customers ask you by phone and email.

Pages that already rank

For most sites the fastest gains in Gemini come from pages already on the first or second page of Google for the question. They are indexed and trusted; they often just bury the answer. Rewriting the opening of those pages so the answer comes first is usually worth more than publishing new ones.

Local questions

For "near me" and service-area questions, Google's answers draw heavily on business listings. A complete and accurate Google Business Profile, with the right categories, services and hours, matters as much as the website. See the Google Business Profile fixes for that side.

Copilot: the Bing side

Copilot builds its web answers from Bing. If Bing has not indexed a page, Copilot cannot cite it, and many sites that have spent years on Google have never looked at Bing at all. The work is quick and mostly one-off.

<!-- Remove these from pages you want Copilot to use, unless they are deliberate -->
<meta name="robots" content="noarchive">
<meta name="bingbot" content="nocache">

Claude: Anthropic's crawlers and fetches

Anthropic documents three crawlers, each with a separate job, and lets site owners control each one by name in robots.txt:

Blocking ClaudeBot is a training decision and does not stop the other two. Blocking Claude-SearchBot or Claude-User reduces how often Claude can cite or read your pages when people ask questions today. A wildcard User-agent: * with Disallow: /, sometimes left over from a staging site, blocks all three.

# robots.txt: let Claude search and fetch; training decided separately
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Claude often fetches a page directly and reads it as text, so pages that depend on scripts to show their content, or that put a consent wall or pop-up in front of it, can come back nearly empty. The test in Step 6 shows you what Claude actually gets.

The fix, step by step

Step 1

Split your prompt results by assistant

Run the same set of customer questions through Gemini, Copilot and Claude, and record for each one whether your site is cited with a link, named without a link, or missing. Put the results in three columns side by side. A gap in one column only points to that assistant's route: Google for Gemini, Bing for Copilot, Anthropic's crawlers for Claude. A gap in all three points to the content itself, so fix that before anything assistant-specific.

Step 2

Cover the follow-up questions Google fans out to

For each main question, list the four to six follow-ups a careful customer would ask next, using People also ask, your Search Console queries and your own enquiries. Check that your page answers each one in a short, clearly headed passage, or links to a page that does. Add the missing answers with real specifics such as costs, timescales, limits and conditions. This gives AI Mode and AI Overviews more places where your page is the best source.

Step 3

Verify the site in Bing Webmaster Tools

Sign in to Bing Webmaster Tools and add the site, importing it from Google Search Console if you already use that. Submit the XML sitemap, then open the indexing reports and note any important pages Bing has excluded or not yet discovered. Use Bing's URL inspection on your five most important answer pages. If they are not indexed, Copilot has nothing to cite, so resolve those exclusions before spending time on anything else for Copilot.

Step 4

Send new and changed pages to Bing with IndexNow

Turn on IndexNow so Bing hears about new and updated pages straight away instead of waiting for a recrawl. On WordPress, several SEO plugins include it as a setting; Cloudflare can send notices from its dashboard; custom sites can post changed URLs to the IndexNow endpoint with a key file placed at the site root. After enabling it, change one page and check in Bing Webmaster Tools that the notice was received.

Step 5

Check the rules Copilot and Claude obey

Read your live robots.txt for lines naming bingbot, Claude-SearchBot, Claude-User or ClaudeBot, and for a blanket Disallow under User-agent: *. Then search page templates for noarchive and nocache robots tags. Search your access logs for bingbot and the Claude agents, and if requests are being refused, check the firewall or CDN bot settings too. Keep only the rules you have chosen deliberately.

Step 6

Ask Claude to read your key pages

With web search turned on in Claude, paste the address of one of your answer pages and ask it to summarise the page and quote its main answer. If Claude says it cannot access the page, or its summary misses the content you care about, the fetch is being blocked or the content is not in the HTML it receives. Check the access log for a Claude-User request at that moment to see the status code it was given.

Step 7

Give each topic one reusable answer paragraph

Under the main heading of each answer page, write one paragraph of two to four sentences that answers the question completely, names your business or product where relevant, and includes the key fact, figure or condition. Avoid references like "as above" that only make sense on the page. All three assistants lift short passages, so one well-made paragraph can serve Gemini, Copilot and Claude at once.

Step 8

Compare the three assistants every month

Run the same prompt set on the same day each month and compare the three columns with the previous month. Look at the direction of travel across many prompts rather than single answers, since each assistant varies from run to run. When one assistant improves and the others do not, the difference usually traces back to indexing on its own route, so check Search Console, Bing Webmaster Tools or your logs first.

Work that helps all three at once

Most of the effort in AEO is not assistant-specific. These tasks improve your chances everywhere and are worth doing before any fine-tuning for a single assistant:

For writing technique in more depth, including before-and-after examples, see fixing content structure for AI extraction.

What not to spend time on

โš ๏ธ Be wary of services promising placement in AI answers. None of these companies sells citations, and no outside party can guarantee them.

Measuring progress for each assistant

Traffic from these assistants is hard to see in analytics. Gemini's contribution through AI Overviews and AI Mode is counted inside the ordinary Web totals in Google Search Console. Copilot and Claude may pass a referrer when someone clicks a source link, but not always, so some visits show up as direct traffic.

That makes prompt-based measurement the main yardstick. The AI Visibility Tracker asks AI assistants the questions your customers ask and records whether your brand is cited, mentioned, recommended or missing, and tracks that over time. Keep the prompt set fixed so month-on-month comparisons mean something, and add new prompts as a separate group rather than mixing them in.

SignalWhere to lookWhat it tells you
Pages indexedSearch Console and Bing Webmaster ToolsWhether Gemini and Copilot can use the page at all.
Crawler requestsServer access logsWhether bingbot and the Claude agents are reaching you, and what status codes they get.
Cited, mentioned or missingRepeated prompts per assistantWhether the content is being chosen once it can be found.
Referral visitsYour analytics, where a referrer is passedWhich pages people click through to from answers.
Questions and answers

1. If I rank well in Google, will I automatically appear in Gemini?

Often, but not automatically. AI Overviews and AI Mode use the Google Search index, so ranking pages are the natural candidates, but Google still chooses passages that answer the specific question well. A page that ranks for a broad term may not contain a direct answer to the narrower question being asked. The Gemini app can also answer from training alone. Ranking gets you considered; a clear, specific answer gets you used.

2. Does Copilot really use Bing, and does that mean Bing rankings matter?

Copilot's web answers are built on Bing search results, so a page Bing has not indexed cannot be cited and a page Bing ranks well has a better chance. Many sites have neglected Bing because its share of ordinary search is smaller than Google's. Verifying the site in Bing Webmaster Tools, submitting a sitemap and enabling IndexNow are small jobs that directly affect whether Copilot can use your pages at all.

3. Should I block ClaudeBot?

That is a decision about training, not about being cited. ClaudeBot collects content that may be used to train future models. Claude-SearchBot and Claude-User are the agents that affect whether Claude can find and read your pages when people ask questions now, and Anthropic lets you control each one separately in robots.txt. Many site owners block the training crawler and allow the search and user agents, but the right answer depends on how you feel about training use.

4. Why does Claude say it cannot access my website?

Usually because a firewall, CDN bot protection or robots.txt rule refused the Claude-User request, or because the page needs JavaScript, a login or a consent click before any content appears. Check your access log for the request at the moment you asked, and note the status code. A 403 or a challenge page points to bot protection; a 200 with little content points to rendering. Allow the agent or serve the main content in the HTML response.

5. Is IndexNow worth setting up for a small site?

Yes, because it takes minutes and costs nothing. IndexNow tells Bing and other participating search engines that a page has changed, so they do not have to wait for a scheduled recrawl to notice. That matters for Copilot, which can only cite what Bing has indexed. Many WordPress SEO plugins and CDNs include it as a switch. Google does not use IndexNow, so keep your XML sitemap up to date for Google as well.

6. Do I need different content for Gemini, Claude and Copilot?

No. All three choose passages that answer the question clearly and specifically, and none of them publishes a preference for a special format. The differences between them are in how they find pages: Google's index, Bing's index, or Anthropic's search and fetch. Fix access and indexing separately for each route, then write one clear page per topic. Trying to maintain separate versions for each assistant would add work without any evidence of benefit.

7. How can I tell how much traffic comes from these assistants?

Only partly. Clicks from AI Overviews and AI Mode are counted inside the Web totals in Google Search Console, not shown separately. Visits from Copilot and Claude appear as referrals when the browser passes a referrer, and as direct traffic when it does not. Because of those gaps, most sites combine analytics with repeated prompt testing, recording whether they are cited or mentioned for a fixed set of customer questions each month.

Compare Gemini, Claude and Copilot

The AI Visibility Tracker asks AI assistants the questions your customers ask and records whether your brand is cited, mentioned, recommended or missing, so you can compare assistants and track progress over time.

Open AEO Fixes โ†’
๐Ÿ’ฌ Got a problem?
Related Guides: All AEO Fixes  ·  Fixing Invisibility in Gemini  ·  Optimising for ChatGPT  ·  Optimising for Perplexity  ·  Learning Hub

About aiwebpageseo

aiwebpageseo.com is a data-driven SEO and AEO (Answer Engine Optimisation) platform providing a free suite of technical SEO tools. Rather than relying on AI-theorised assumptions, the platform analyses live URL performance, delivering objective diagnostics, page speed metrics, CLS debugging, and site crawl data alongside actionable technical tutorials.