AI Crawlers Blocked
An AI assistant can only quote, summarise or link to a page that its crawler was allowed to fetch. Many sites turn these crawlers away without meaning to: a catch-all robots.txt rule written years ago, a security plugin that treats every unfamiliar bot as an attacker, or a CDN switch that blocks "AI bots" with one click.
Visitors never notice, because browsers are let through. This guide shows how to find which layer is refusing which crawler, how to write robots.txt rules that say exactly what you intend, and how to confirm in your logs that the crawlers you chose are now getting real pages.
Four places a crawler can be refused
When the Agent Readiness Audit reports that an AI crawler cannot reach a page, the refusal comes from one of four places. They behave differently, so it is worth knowing which one you are dealing with before changing anything.
| Layer | What the crawler experiences | Typical cause |
|---|---|---|
| robots.txt | It reads the file, finds a disallow rule for its name or for *,
and never requests the page. | An old Disallow: / for all bots, a copied block list,
a staging robots.txt pushed to production. |
| CDN or edge firewall | It requests the page and receives a 403, a JavaScript challenge or a CAPTCHA instead of HTML. | An "AI bots" or "bot fight" toggle, a managed rule set, a country block. |
| Web server or security plugin | A 403, 406 or 429 from your own server, sometimes after a few successful requests. | Wordfence, ModSecurity, fail2ban, a user agent deny list in .htaccess
or nginx. |
| Page-level directives | The page is fetched but marked as off limits for indexing or snippets. | noindex
or nosnippet in a meta robots tag or an X-Robots-Tag header. |
The first layer is polite: a crawler that honours robots.txt simply stays away and leaves no trace
in your logs beyond a request for /robots.txt. The second and third are forceful: the crawler
arrives and is refused, which shows up as error codes. The fourth is a different kind of problem, because
the page is fetched but the site has asked for it not to be used, and each provider decides for itself
how its systems treat those tags.
/robots.txt
and then nothing else, robots.txt is the reason. If it requests pages and gets 403s, look at the firewall
or server instead.The crawler names the providers publish
Only use names that the providers themselves document. Lists copied from forums often contain retired names, invented ones and bots that never read robots.txt. The table below covers the main published names as of September 2026. Providers do rename and add crawlers, so check each provider's own documentation before relying on it.
| Provider | Name | What the provider says it is for |
|---|---|---|
| OpenAI | GPTBot | Collecting content that may be used to train its models. |
| OpenAI | OAI-SearchBot | Finding and surfacing sites in ChatGPT search results. |
| OpenAI | ChatGPT-User | Fetching a page because a ChatGPT user asked for it. OpenAI says robots.txt rules may not apply to these user-initiated visits. |
| Anthropic | ClaudeBot | Collecting web content to improve its models. |
| Anthropic | Claude-SearchBot | Improving the quality and relevance of search results for Claude users. |
| Anthropic | Claude-User | Fetching a page when a person asks Claude a question that needs it. Anthropic says its bots honour robots.txt. |
| Perplexity | PerplexityBot | Building the index that Perplexity links to in its answers; Perplexity says it is not used to train foundation models. |
| Perplexity | Perplexity-User | Fetching a page for a user's question; Perplexity says it generally ignores robots.txt because a person requested it. |
Google-Extended | A robots.txt token only, not a separate crawler. It controls whether content Googlebot collects may be used for Gemini models; Google says it does not affect Google Search. | |
| Apple | Applebot-Extended | A token only. It controls whether content Applebot collects may be used to train Apple's models. |
| Common Crawl | CCBot | Building an open web archive that many research and AI projects use. |
| Meta | Meta-ExternalAgent | Collecting content for Meta's AI work, according to Meta's crawler documentation. |
Two points catch people out. First, Google's AI Overviews and AI Mode are part of Google Search, so
they depend on ordinary Googlebot access, not on Google-Extended. Blocking
Googlebot to keep out of AI features would remove you from Search entirely. Second, Microsoft's Copilot
draws on the Bing index, so a rule against bingbot has AI consequences as well as search
ones.
anthropic-ai and Claude-Web still
appear in many robots.txt files. They are not in Anthropic's current documentation, so rules for them
do nothing useful. Leaving them in is harmless, but do not assume they cover the current bots.How crawlers read robots.txt groups
Most unintended AI blocks come from misunderstanding how groups work. The rules are set out in RFC 9309, the robots exclusion standard, and the major crawlers follow it.
- A crawler picks one group. It looks for a group whose
User-agentline matches its own name. If it finds one, it follows only that group and ignores the*group completely. If it finds none, it follows*. - Name matching ignores case.
gptbotandGPTBotare the same, but the name must be the product token, not the whole user agent string. - Several names can share a group. Stack
User-agentlines directly above one set of rules. A blank line between them can split the group in some parsers, so keep them together. - The most specific path wins. Within a group, the longest matching
AlloworDisallowpath applies. On a tie,Allowwins. - Each host has its own file.
www.example.co.uk,example.co.ukandshop.example.co.ukare each governed by the robots.txt at their own root.
The first rule causes a classic mistake. Imagine a site that blocks its admin area for everyone, and later adds a group to welcome an AI crawler:
User-agent: *
Disallow: /admin/
Disallow: /basket/
User-agent: OAI-SearchBot
Allow: /OAI-SearchBot now follows only its own group, so it is allowed into /admin/ and /basket/.
The reverse is more common and more damaging: a site adds a named group containing one harmless rule,
and the crawler stops obeying the * group that held all the real restrictions. Every named
group must repeat the rules that should still apply.
Three robots.txt policies, written correctly
Pick the policy that matches your decision, then adapt the paths. Each example is complete, so a crawler that matches a named group still gets the private-area rules.
Open to search and assistant crawlers, closed to training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Disallow: /admin/
Disallow: /basket/
Disallow: /search?
Sitemap: https://www.example.co.uk/sitemap.xmlOAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-initiated fetchers have no named group
here, so they follow * and can read everything except the private areas.
Open to all AI crawlers
User-agent: *
Disallow: /admin/
Disallow: /basket/
Sitemap: https://www.example.co.uk/sitemap.xmlNothing else is needed. Adding Allow: / groups for each AI crawler adds no permission
they did not already have and risks the group trap described above.
Closed to AI crawlers, open to ordinary search
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Disallow: /admin/This keeps Googlebot and bingbot, so ordinary search listings continue. Be aware that user-initiated fetchers may still visit when a person pastes your link into an assistant, because some providers say robots.txt does not apply to those requests. If you need a hard stop, it has to be enforced at the firewall.
The firewall and CDN layer
A robots.txt file that says yes counts for nothing if the edge says no. This layer has become the most common cause of surprise blocks, because it is often configured by a hosting company or an agency rather than by whoever looks after the content.
- One-click AI blocking. Several CDNs offer a setting to block known AI crawlers, and at least one major provider has enabled it by default on new domains. Look in the security or bot section of the dashboard for anything mentioning AI crawlers, AI scrapers or crawl control.
- Bot fight and challenge modes. Features that send suspected bots a JavaScript challenge stop every crawler that does not run scripts. The crawler records a challenge page, not your content.
- Security plugins. WordPress security plugins can block user agents they consider unknown, or rate-limit any client that requests many pages quickly, which is exactly what a crawler does.
- Country and ASN rules. AI crawlers mostly run from large cloud providers outside the UK. A rule that blocks traffic from other countries, or from cloud hosting networks, catches them.
- Hand-written deny lists. Search nginx configuration and
.htaccessfiles forGPTBot,ClaudeorBytespider. Block lists are often pasted in during a load problem and forgotten.
When you add an allow rule, match on the provider's verified identity rather than the user agent string alone. Anyone can send a request claiming to be GPTBot. OpenAI and Perplexity publish the IP ranges their crawlers use, and several CDNs maintain their own lists of verified bots, so an allow rule based on those cannot be abused by a scraper that borrows the name.
Reading the evidence
Two quick checks tell you most of what you need. First, request a page while presenting a crawler's product token as the user agent, and look only at the status line:
for ua in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do
printf "%-18s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.co.uk/services/
doneA mix of 200s and 403s shows a user agent rule in the firewall or server. All 200s does not prove crawlers are welcome, because rules based on IP ranges or verified-bot lists will treat your own machine differently from the real crawler. That is why the second check matters: search the access logs for the real visits.
grep -iE "gptbot|oai-searchbot|claudebot|claude-searchbot|claude-user|perplexitybot" access.log \
| awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -20| What the log shows | What it means |
|---|---|
Only requests for /robots.txt | The crawler read your rules and decided it may not go further. |
| 403 or 406 responses | Your server or a security plugin refused it. |
| 429 responses | Rate limiting. Raise the limit for verified crawlers or add a Crawl-delay
for crawlers that support it. |
| 503 responses | Overload or a challenge page served with an error code. |
| No lines at all | Blocked at the CDN before reaching the server, or the crawler has not visited yet. Check the CDN's own logs. |
| 200 responses for pages | Access is working. Any remaining audit findings are about content, not permission. |
The fix, step by step
Step 1
List the crawlers you want and the ones you do not
Write down a short decision before editing anything. For each provider, decide
whether you want its search or answer crawler, its training crawler and its user-initiated fetcher. A
shop that wants to be recommended will usually allow OAI-SearchBot, Claude-SearchBot
and PerplexityBot, and may choose either way on GPTBot, ClaudeBot
and CCBot. Take the names only from each provider's current documentation, not from an old
block list.
Step 2
Read the live robots.txt group by group
Open /robots.txt on every host you use, including the www and non-www versions
and any shop or blog subdomain. For each crawler on your list, find the one group it will obey: its own
named group if there is one, otherwise the * group. Note what that group allows. Pay special
attention to named groups that contain only one rule, because the crawler ignores everything in *
once it has its own group.
Step 3
Rewrite the rules so each crawler gets the intended answer
Put crawlers you want to exclude in a single group with Disallow: /,
stacking their User-agent lines together without blank lines. Let crawlers you welcome fall
through to the * group, and keep your private-area rules there. Remove pointless Allow: /
groups and retired names. Keep the Sitemap line. If your platform generates robots.txt,
make the change in its settings or template so an update does not overwrite it.
Step 4
Clear the firewall, CDN and plugin blocks
Open your CDN dashboard and look for AI crawler blocking, bot fight modes, managed challenge rules and country or network blocks. Then check your security plugin and server configuration for user agent deny lists and aggressive rate limits. For each crawler you chose to allow, add a narrow exception based on the provider's published IP ranges or the CDN's verified bot list, and leave challenges in place for unverified traffic.
Step 5
Test each user agent from outside your network
Run the Agent Readiness Audit again and compare its crawler results with your decision
list. Then use curl -A with each product token against a few key pages and note the status
code of each response. Expect 200 for crawlers you allowed. Remember that a clean curl result
cannot prove that IP-based rules are correct, because your own address is not in a provider's range,
so do not stop here.
Step 6
Confirm real visits in the access logs
Over the following days, search your server or CDN logs for each crawler name and group the
results by status code. You want to see page requests returning 200, not just requests for
/robots.txt followed by silence. If a crawler you allowed still receives 403, 429 or 503,
trace the request in the CDN event log to find which rule fired, adjust it, and check again.
Edge cases worth knowing
- robots.txt that errors. A missing file (404) means everything is allowed. A server
error (5xx) is different: Google treats it as a temporary full block, and other crawlers may be just
as cautious. Make sure
/robots.txtreturns a fast 200. - robots.txt that redirects. Crawlers follow a few redirects, but a redirect to an
HTML page or a login screen is read as an invalid file. Serve it directly as
text/plain. - File size. Google reads only the first 500 KiB. Very long block lists pasted from the web can push real rules past that point.
- Crawl-delay. Google ignores it, but Anthropic documents support for it. If load is the reason for a block, a delay for the crawlers that respect it is a gentler answer than a ban.
- Staging and preview sites. Keep them behind a password rather than relying on robots.txt,
and make sure their
Disallow: /file is never deployed to the live site. - Noindex is not a block. A
noindextag stops a page being listed but the crawler still fetches it. A robots.txt disallow stops the fetch, so the crawler never sees the noindex at all. Do not use both on the same URL if you want the noindex to be read.
Keeping access the way you set it
- Keep robots.txt in version control with a short comment at the top recording the policy and the date it was decided, so the next person knows the blocks are deliberate.
- Review after infrastructure changes. A new CDN, a hosting move or a new security plugin can undo months of careful rules in a minute.
- Watch for new names. Providers add crawlers. A quarterly look at each provider's documentation, and at unfamiliar agents in your logs, keeps the policy current.
- Alert on sudden changes. A log alert when a chosen crawler starts receiving mostly 403s will tell you about a block long before a drop in AI referrals would.
For the related robots.txt problems that affect ordinary search crawlers, see the robots.txt AI blocks guide, and once access is sorted, move on to content extractability.
1. Which AI crawlers should a small business allow?
There is no single right answer, because it depends on whether you want your content used to train models as well as to answer questions. Many small businesses allow the search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, because those are the ones that let assistants find, cite and link to their pages. The training crawlers, such as GPTBot, ClaudeBot and CCBot, are a separate choice you can make either way without affecting the search ones.
2. Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended is a robots.txt token that controls whether content Google has already crawled may be used for its Gemini models. Google says it does not affect Google Search, and AI Overviews and AI Mode are part of Search. They depend on ordinary Googlebot access. The only way to stay out of them through robots.txt would be to block Googlebot itself, which removes you from Google Search results too, so almost no business should do that.
3. Why does the audit say GPTBot is blocked when my robots.txt allows everything?
Because robots.txt is only one of the layers. A CDN setting that blocks AI crawlers, a bot challenge page, a security plugin or a rate limit can still refuse the request with a 403, 429 or 503 status. The crawler is technically allowed but practically turned away. Search your access logs for GPTBot to see the status codes it received, then check your CDN's bot and security settings, where the block usually turns out to be.
4. Can I trust a request that says it is ClaudeBot or GPTBot?
Not on the user agent string alone, since any scraper can copy it. For allow rules and for log analysis, verify the source. OpenAI and Perplexity publish the IP address ranges their crawlers use, and several CDNs keep their own verified bot lists that check identity automatically. Base firewall exceptions on those rather than on the name. That way you welcome the real crawler without opening a door that anyone could walk through by changing one header.
5. Do AI crawlers really obey robots.txt?
The major providers say their automated crawlers do, and you can see it in logs when a crawler reads robots.txt and then stops. User-initiated fetchers are different. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for that specific page. Scrapers that do not identify themselves ignore robots.txt entirely. If you need a guaranteed block, enforce it at the firewall as well.
6. Should I still have rules for anthropic-ai and Claude-Web?
They do no harm, but they no longer do anything useful. Anthropic's current documentation lists ClaudeBot, Claude-SearchBot and Claude-User, and a rule for an older name does not cover any of them. If your aim is to block or allow Anthropic's crawlers, write the rule for the current names. Keeping the old lines is fine if you prefer a belt-and-braces file, as long as nobody mistakes them for the rules that actually apply today.
7. How long after unblocking will AI crawlers come back?
There is no published timetable. Crawlers re-read robots.txt regularly, often within a day, so an allowed crawler can start fetching pages soon after the change. How often it returns depends on the provider and on how much it values your site. A firewall change takes effect immediately for new requests. Watch your logs over a week or two rather than expecting an instant change in how often assistants mention you.
The Agent Readiness Audit requests your page with real AI crawler user agents and checks robots.txt and reports what stops each one reading it, so you can see which crawlers get through and which are turned away.
Open Agent Readiness Fixes →