Is your site AI agent ready?

16 October 2026 · By John Carlsson

Your site is AI agent ready when the crawlers and assistants you want can reach your pages through robots.txt and read the main content as text. OpenAI and Anthropic each name their search, user and training agents. For Google AI Overviews and AI Mode, a page only needs to be indexed and eligible for a snippet, with no special markup or AI text files.

Which AI crawlers and agents visit your site?

The main AI companies publish the names of their user agents, so you can decide which ones to allow. OpenAI runs three. OAI-SearchBot is used to surface websites in ChatGPT search features, and OpenAI says sites that block it will not be shown in ChatGPT search answers, although they can still appear as navigational links. ChatGPT-User visits a page when a person asks ChatGPT or a custom GPT a question that needs it. GPTBot crawls content that may be used to train OpenAI foundation models.

Anthropic also runs three. ClaudeBot collects web content for model development. Claude-User fetches pages when a Claude user asks a question that needs them. Claude-SearchBot crawls content to improve the quality of search results. Anthropic notes that disabling Claude-User or Claude-SearchBot may reduce your visibility in answers that use web search.

Being ready for agents therefore means deciding which of these you want. Many site owners allow the search and user agents, which send people to their pages, and make a separate choice about the training crawlers.

How do you allow or block them in robots.txt?

Both companies use robots.txt as the control. Anthropic says its bots honour industry standard robots.txt directives and also support the non-standard Crawl-delay extension. A file that allows the search agents but opts out of training could look like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

There are two points to watch. First, OpenAI says that because ChatGPT-User actions are started by a person, robots.txt rules may not apply to them. Second, Anthropic advises against relying on IP blocking, because it may not work reliably and can stop its bots from reading your robots.txt file at all. If you want to confirm a request really came from Anthropic, it publishes a list of its IP addresses for that purpose. Remember that robots.txt rules apply per host, so each subdomain needs its own file.

What does Google need for AI Overviews and AI Mode?

Google says there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimisations are necessary. A page must be indexed and eligible to be shown in Google Search with a snippet. Google states that you do not need new machine readable files, AI text files or markup, and that there is no special schema.org structured data to add. In June 2026 Google also clarified that llms.txt files are not needed for Search and do not affect rankings.

The normal SEO basics still apply. Googlebot must be allowed to crawl the page in robots.txt, and the important content should be available as text rather than locked inside images or scripts. Because AI features are part of Search, the robots.txt rules for Googlebot are what control them. Google-Extended is a separate control that covers AI training and grounding in some of Google's other systems, not Search.

How can you control what AI features show from your pages?

Google lists four controls that limit what Search shows from a page, including in AI features:

nosnippet, which stops a text snippet being shown
data-nosnippet, which hides a specific part of a page
max-snippet, which limits the snippet length
noindex, which keeps the page out of Search altogether

If content still appears after you add a control, check with the URL Inspection tool that Googlebot can see it, then allow time for recrawling, which Google says can take from several days to several months. Traffic from AI features is counted within the Web search type in the Search Console Performance report, alongside your other Google Search traffic.

A quick readiness check covers four things: your robots.txt rules for each agent, whether your key pages are indexed, whether the main content is plain text in the HTML, and whether any snippet controls are hiding more than you intended.

Sources

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses separate user agents. GPTBot crawls content that may be used to train its foundation models, while OAI-SearchBot is the agent used to surface websites in ChatGPT search features. You can disallow GPTBot and still allow OAI-SearchBot. OpenAI says that sites blocking OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links.

Do I need an llms.txt file for Google AI Overviews?

No. Google says you do not need new machine readable files, AI text files or special markup to appear in AI Overviews or AI Mode. In June 2026 it also clarified that llms.txt files are not needed for Search and do not affect rankings. The page must be indexed and eligible to show with a snippet, and normal SEO best practices apply.