Optimising for Gemini, Claude and Copilot
ChatGPT gets most of the attention, but plenty of your customers ask Gemini inside Google, Copilot inside Windows and Edge, or Claude. Each of these assistants finds sources through a different route: Gemini leans on Google Search, Copilot on Bing, and Claude on Anthropic's own crawlers and search.
That means a site can do well in one and be absent from the others. This guide explains each route, what you can influence in each, and a single plan of work that improves all three without tripling the effort.
Three assistants, three routes to your site
All three assistants write answers from a mix of what their model learned in training and what they fetch from the web at the time of the question. The part you can influence quickly is the fetching, and that is where they differ.
| Assistant | Where live sources come from | Crawler names to know | Your main lever |
|---|---|---|---|
| Gemini, AI Overviews, AI Mode | The Google Search index. | Googlebot;
Google-Extended is a robots.txt token, not a crawler. | Google Search Console and how well your pages answer the question. |
| Microsoft Copilot | The Bing index. | bingbot | Bing Webmaster Tools, IndexNow and Bing's own ranking. |
| Claude | Anthropic's web search, plus pages fetched when a user asks. | Claude-SearchBot,
Claude-User, ClaudeBot (training). | Crawler access and pages that read well when fetched directly. |
The overlap is larger than the table suggests. All three reward the same thing once they have your page: a clear, specific answer near the top, written so a sentence or two can be lifted out and still make sense. The differences are mostly about getting found in the first place.
Gemini: competing inside Google Search
Google's own guidance on AI features says there are no extra technical requirements for appearing in AI Overviews or AI Mode beyond being indexed and eligible to show a snippet, and that normal Search best practice applies. That is useful to know, because it rules out a lot of paid advice about special AI tags. What it leaves is the quality and coverage of your content.
Fan-out questions
Google has described AI Mode as breaking a question into several related searches, running them together and combining what it finds. A person asking "Is it worth fitting a heat pump in a 1930s semi?" may trigger searches about running costs, insulation, grants, noise and installation time. A page that answers only the headline question competes for one of those searches. A page, or a small cluster of linked pages, that answers the follow-ups too has more chances to be used.
You can find likely follow-ups in the "People also ask" boxes for your main question, in Search Console queries that already bring impressions to the page, and in the questions customers ask you by phone and email.
Pages that already rank
For most sites the fastest gains in Gemini come from pages already on the first or second page of Google for the question. They are indexed and trusted; they often just bury the answer. Rewriting the opening of those pages so the answer comes first is usually worth more than publishing new ones.
Local questions
For "near me" and service-area questions, Google's answers draw heavily on business listings. A complete and accurate Google Business Profile, with the right categories, services and hours, matters as much as the website. See the Google Business Profile fixes for that side.
Copilot: the Bing side
Copilot builds its web answers from Bing. If Bing has not indexed a page, Copilot cannot cite it, and many sites that have spent years on Google have never looked at Bing at all. The work is quick and mostly one-off.
- Bing Webmaster Tools. Verify the site, and if you already use Google Search Console you can import the verified sites from it rather than verifying again. Submit your XML sitemap and check the indexing reports for pages Bing has excluded.
- IndexNow. Bing supports IndexNow, a simple protocol that lets a site tell search engines a page has been added, changed or removed. Many CMS plugins and CDNs can send IndexNow notices automatically, which shortens the wait for Bing to notice an update.
- URL Inspection in Bing. Bing Webmaster Tools has its own URL inspection, which shows whether a page is indexed and whether Bing found problems with it. Use it on the pages you most want Copilot to cite.
- Robots meta tags. When Microsoft launched chat answers in Bing in 2023, it said
the
noarchiveandnocacherobots meta tags would limit how page content is used in those answers. Search your templates for either tag; many sites addednoarchivelong ago for unrelated reasons. Check Bing's current webmaster guidance before relying on the exact behaviour.
<!-- Remove these from pages you want Copilot to use, unless they are deliberate -->
<meta name="robots" content="noarchive">
<meta name="bingbot" content="nocache">
Claude: Anthropic's crawlers and fetches
Anthropic documents three crawlers, each with a separate job, and lets site owners control each one by name in robots.txt:
ClaudeBotcollects content that may be used to train future models.Claude-SearchBotcrawls pages to improve the results Claude uses when it searches the web.Claude-Userfetches a page when a person using Claude asks about it or when Claude needs it to answer them.
Blocking ClaudeBot is a training decision and does not stop the other two. Blocking Claude-SearchBot
or Claude-User reduces how often Claude can cite or read your pages when people ask questions
today. A wildcard User-agent: * with Disallow: /, sometimes left over from
a staging site, blocks all three.
# robots.txt: let Claude search and fetch; training decided separately
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
Claude often fetches a page directly and reads it as text, so pages that depend on scripts to show their content, or that put a consent wall or pop-up in front of it, can come back nearly empty. The test in Step 6 shows you what Claude actually gets.
The fix, step by step
Step 1
Split your prompt results by assistant
Run the same set of customer questions through Gemini, Copilot and Claude, and record for each one whether your site is cited with a link, named without a link, or missing. Put the results in three columns side by side. A gap in one column only points to that assistant's route: Google for Gemini, Bing for Copilot, Anthropic's crawlers for Claude. A gap in all three points to the content itself, so fix that before anything assistant-specific.
Step 2
Cover the follow-up questions Google fans out to
For each main question, list the four to six follow-ups a careful customer would ask next, using People also ask, your Search Console queries and your own enquiries. Check that your page answers each one in a short, clearly headed passage, or links to a page that does. Add the missing answers with real specifics such as costs, timescales, limits and conditions. This gives AI Mode and AI Overviews more places where your page is the best source.
Step 3
Verify the site in Bing Webmaster Tools
Sign in to Bing Webmaster Tools and add the site, importing it from Google Search Console if you already use that. Submit the XML sitemap, then open the indexing reports and note any important pages Bing has excluded or not yet discovered. Use Bing's URL inspection on your five most important answer pages. If they are not indexed, Copilot has nothing to cite, so resolve those exclusions before spending time on anything else for Copilot.
Step 4
Send new and changed pages to Bing with IndexNow
Turn on IndexNow so Bing hears about new and updated pages straight away instead of waiting for a recrawl. On WordPress, several SEO plugins include it as a setting; Cloudflare can send notices from its dashboard; custom sites can post changed URLs to the IndexNow endpoint with a key file placed at the site root. After enabling it, change one page and check in Bing Webmaster Tools that the notice was received.
Step 5
Check the rules Copilot and Claude obey
Read your live robots.txt for lines naming bingbot, Claude-SearchBot,
Claude-User or ClaudeBot, and for a blanket Disallow under User-agent: *.
Then search page templates for noarchive and nocache robots tags. Search your
access logs for bingbot and the Claude agents, and if requests are being refused, check the firewall
or CDN bot settings too. Keep only the rules you have chosen deliberately.
Step 6
Ask Claude to read your key pages
With
web search turned on in Claude, paste the address of one of your answer pages and ask it to summarise
the page and quote its main answer. If Claude says it cannot access the page, or its summary misses the
content you care about, the fetch is being blocked or the content is not in the HTML it receives. Check
the access log for a Claude-User request at that moment to see the status code it was given.
Step 7
Give each topic one reusable answer paragraph
Under the main heading of each answer page, write one paragraph of two to four sentences that answers the question completely, names your business or product where relevant, and includes the key fact, figure or condition. Avoid references like "as above" that only make sense on the page. All three assistants lift short passages, so one well-made paragraph can serve Gemini, Copilot and Claude at once.
Step 8
Compare the three assistants every month
Run the same prompt set on the same day each month and compare the three columns with the previous month. Look at the direction of travel across many prompts rather than single answers, since each assistant varies from run to run. When one assistant improves and the others do not, the difference usually traces back to indexing on its own route, so check Search Console, Bing Webmaster Tools or your logs first.
What not to spend time on
- Special AI meta tags. None of the three assistants documents a tag that increases your chances of being cited. Tags you find recommended online either do nothing or are standard robots rules that can only restrict use.
- Allowing training crawlers to "get cited". Training crawlers such as
ClaudeBotand the Google-Extended token affect future models, not the live search that produces most links. Decide on training separately and on its merits. - Rewriting for each assistant. Separate versions of a page for Gemini, Claude and Copilot are not possible and not needed. One clear page serves all three.
- Chasing a single answer. An answer that includes you today may not tomorrow. Judge progress over a month of repeated prompts.
- Keyword repetition. Repeating a phrase does not make a passage more quotable. Specific, correct facts do.
Measuring progress for each assistant
Traffic from these assistants is hard to see in analytics. Gemini's contribution through AI Overviews and AI Mode is counted inside the ordinary Web totals in Google Search Console. Copilot and Claude may pass a referrer when someone clicks a source link, but not always, so some visits show up as direct traffic.
That makes prompt-based measurement the main yardstick. The AI Visibility Tracker asks AI assistants the questions your customers ask and records whether your brand is cited, mentioned, recommended or missing, and tracks that over time. Keep the prompt set fixed so month-on-month comparisons mean something, and add new prompts as a separate group rather than mixing them in.
| Signal | Where to look | What it tells you |
|---|---|---|
| Pages indexed | Search Console and Bing Webmaster Tools | Whether Gemini and Copilot can use the page at all. |
| Crawler requests | Server access logs | Whether bingbot and the Claude agents are reaching you, and what status codes they get. |
| Cited, mentioned or missing | Repeated prompts per assistant | Whether the content is being chosen once it can be found. |
| Referral visits | Your analytics, where a referrer is passed | Which pages people click through to from answers. |
1. If I rank well in Google, will I automatically appear in Gemini?
Often, but not automatically. AI Overviews and AI Mode use the Google Search index, so ranking pages are the natural candidates, but Google still chooses passages that answer the specific question well. A page that ranks for a broad term may not contain a direct answer to the narrower question being asked. The Gemini app can also answer from training alone. Ranking gets you considered; a clear, specific answer gets you used.
2. Does Copilot really use Bing, and does that mean Bing rankings matter?
Copilot's web answers are built on Bing search results, so a page Bing has not indexed cannot be cited and a page Bing ranks well has a better chance. Many sites have neglected Bing because its share of ordinary search is smaller than Google's. Verifying the site in Bing Webmaster Tools, submitting a sitemap and enabling IndexNow are small jobs that directly affect whether Copilot can use your pages at all.
3. Should I block ClaudeBot?
That is a decision about training, not about being cited. ClaudeBot collects content that may be used to train future models. Claude-SearchBot and Claude-User are the agents that affect whether Claude can find and read your pages when people ask questions now, and Anthropic lets you control each one separately in robots.txt. Many site owners block the training crawler and allow the search and user agents, but the right answer depends on how you feel about training use.
4. Why does Claude say it cannot access my website?
Usually because a firewall, CDN bot protection or robots.txt rule refused the Claude-User request, or because the page needs JavaScript, a login or a consent click before any content appears. Check your access log for the request at the moment you asked, and note the status code. A 403 or a challenge page points to bot protection; a 200 with little content points to rendering. Allow the agent or serve the main content in the HTML response.
5. Is IndexNow worth setting up for a small site?
Yes, because it takes minutes and costs nothing. IndexNow tells Bing and other participating search engines that a page has changed, so they do not have to wait for a scheduled recrawl to notice. That matters for Copilot, which can only cite what Bing has indexed. Many WordPress SEO plugins and CDNs include it as a switch. Google does not use IndexNow, so keep your XML sitemap up to date for Google as well.
6. Do I need different content for Gemini, Claude and Copilot?
No. All three choose passages that answer the question clearly and specifically, and none of them publishes a preference for a special format. The differences between them are in how they find pages: Google's index, Bing's index, or Anthropic's search and fetch. Fix access and indexing separately for each route, then write one clear page per topic. Trying to maintain separate versions for each assistant would add work without any evidence of benefit.
7. How can I tell how much traffic comes from these assistants?
Only partly. Clicks from AI Overviews and AI Mode are counted inside the Web totals in Google Search Console, not shown separately. Visits from Copilot and Claude appear as referrals when the browser passes a referrer, and as direct traffic when it does not. Because of those gaps, most sites combine analytics with repeated prompt testing, recording whether they are cited or mentioned for a fixed set of customer questions each month.
The AI Visibility Tracker asks AI assistants the questions your customers ask and records whether your brand is cited, mentioned, recommended or missing, so you can compare assistants and track progress over time.
Open AEO Fixes โ