Fixing robots.txt Rules That Block AI Crawlers From Your llms.txt

Why the auditor treats this as a conflict

robots.txt and llms.txt do different jobs. robots.txt, defined in RFC 9309, is the file crawlers that follow the Robots Exclusion Protocol read before fetching anything; it can refuse them paths. llms.txt, a proposal published by Jeremy Howard in 2024, grants and refuses nothing; it only recommends. When the two disagree, robots.txt wins for every crawler that respects it. A page listed in llms.txt but disallowed for a crawler is simply never fetched by that crawler, however well you described it.

So the auditor checks three things together:

1. Can AI crawlers fetch /llms.txt and /llms-full.txt at all? A rule that blocks the file makes it invisible to any crawler that obeys robots.txt.
2. Can they fetch the pages the file links to? A file whose best links sit in a disallowed folder recommends pages the reader is told not to open.
3. Is the refusal what you meant? A deliberate choice to keep training crawlers out is a legitimate policy, not a fault. An accidental block of search and user-requested fetchers usually is a fault.

Be clear about what fixing this achieves. As far as we know, no major AI provider has confirmed that its crawlers or answer products read llms.txt. The value of removing the contradiction is that the crawlers you have chosen to welcome can reach the pages you consider most important, whether or not they ever look at llms.txt. If you have decided to refuse AI crawlers altogether, the consistent fix is the opposite one: keep robots.txt as it is and question whether you need an llms.txt at all.

The user agents worth testing

Test only names that providers publish in their own documentation. Block lists copied from forums are full of names that are retired, misspelt or never existed, and testing against those tells you nothing. The AI crawler access guide lists what each provider says its crawlers are for; for this check, group them by the role they would play for a reader of llms.txt.

Providers add and rename crawlers, so check their documentation when you make changes. Old names such as anthropic-ai and Claude-Web are no longer in Anthropic's documentation; rules for them neither help nor harm, but they do not cover the current bots.

Rules that catch llms.txt by accident

Almost nobody writes Disallow: /llms.txt on purpose. The file gets caught by broader rules. These are the patterns that turn up most often when the auditor reports a conflict.

The last row surprises people. RFC 9309 says that if robots.txt cannot be fetched because of a server error, a crawler should assume it may crawl nothing, whereas a 404 for robots.txt means it may crawl everything. A robots.txt that intermittently returns 500 can therefore block every crawler that follows the standard, AI or otherwise, without a single disallow rule in sight. Check the status code, not just the content.

The narrowest fix for a wildcard rule that catches llms.txt is an explicit allow. Within a group, the most specific matching rule wins, so a longer Allow path overrides a shorter Disallow pattern:

User-agent: *
Disallow: /*.txt$
Allow: /llms.txt
Allow: /llms-full.txt

Better still, ask why the .txt rule exists. robots.txt is public, so a rule that names files you want hidden advertises them; files that must stay private need removing from the web root or protecting with a login, not a disallow line.

Where the offending rule is really coming from

The robots.txt you see in your editor is not always the one crawlers receive. Before changing anything, find out what produces the live file.

• A physical file in the web root, edited by hand or deployed from a repository.
• A CMS or SEO plugin generating it on the fly. WordPress creates a virtual robots.txt when no physical file exists, and SEO plugins such as Yoast SEO, Rank Math and All in One SEO let you edit it from the dashboard. A physical file, if present, overrides the virtual one.
• A CDN or hosting layer adding lines. Some CDNs, Cloudflare among them, offer a setting that manages robots.txt for you and can add disallow rules for AI crawlers. If rules appear that are in neither your file nor your CMS, look there.
• A platform default on hosted builders and shop platforms, where robots.txt is generated from templates or settings rather than a file you edit directly.
• A different host. Each host name has its own robots.txt. If llms.txt on www.example.co.uk links to docs.example.co.uk or shop.example.co.uk, those pages are governed by the robots.txt on those hosts.

curl -s -D - https://www.example.co.uk/robots.txt -o robots-live.txt
curl -s -D - https://docs.example.co.uk/robots.txt -o robots-docs.txt

The -D - option prints the response headers, so you can see the status code and any CDN headers alongside saving the body. Compare the saved files with what your CMS shows; any difference points to a layer in between.

Testing every llms.txt URL against every agent

Checking by eye works for a short file and a simple robots.txt, but wildcards and group selection make it easy to misread. This script fetches llms.txt, collects every linked URL plus the two llms files themselves, loads the robots.txt for each host and reports which agents are refused which URLs. It uses protego, a robots.txt parser that supports the * and $ wildcards. Python's built-in urllib.robotparser does not handle those wildcards, so it can give wrong answers for modern files.

# pip install requests protego
import re, requests
from urllib.parse import urlsplit
from protego import Protego

SITE = "https://www.example.co.uk"
AGENTS = ["OAI-SearchBot", "ChatGPT-User", "GPTBot",
          "Claude-SearchBot", "Claude-User", "ClaudeBot",
          "PerplexityBot", "Perplexity-User", "CCBot"]

index = requests.get(SITE + "/llms.txt", timeout=20).text
urls = [SITE + "/llms.txt", SITE + "/llms-full.txt"]
urls += re.findall(r"\]\((https?://[^)\s]+)\)", index)

parsers = {}
for url in dict.fromkeys(urls):
    host = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    if host not in parsers:
        r = requests.get(host + "/robots.txt", timeout=20)
        print(host, "robots.txt status", r.status_code)
        parsers[host] = Protego.parse(r.text if r.status_code == 200 else "")
    refused = [a for a in AGENTS if not parsers[host].can_fetch(url, a)]
    if refused:
        print("BLOCKED", url, "->", ", ".join(refused))

Read the status line for each host first. The script treats any non-200 robots.txt as empty, which matches a 404 but is too generous for a 5xx: if a host reports a server error, fix that before trusting anything else in the output. Google-Extended and Applebot-Extended are left out because they control use of content rather than fetching; review their groups by eye.

The fix, step by step

A worked example

A fictional software company, Fenwick Rota, publishes an llms.txt whose main section links to its help centre under /help/. Its robots.txt had grown over several years:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

User-agent: *
Disallow: /app/
Disallow: /help/v1/
Disallow: /*.txt$

The test script reported that every agent listed in the first group was refused every URL, and that all other agents were refused /llms.txt by the .txt rule. The company's agreed policy was: no training, yes to AI search and to user-requested fetches. The rewrite:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /app/
Disallow: /help/v1/
Allow: /llms.txt
Allow: /llms-full.txt

Sitemap: https://www.example.com/sitemap.xml

OAI-SearchBot, ChatGPT-User and PerplexityBot now fall through to the * group and can read the help centre and both llms files. The .txt rule was removed after the team confirmed it had been added years ago to hide a changelog that is now public anyway; the two Allow lines are kept as a guard in case someone adds a similar rule again. The retired /help/v1/ section remains refused, and the three links to it were removed from llms.txt at the same time.

Keeping robots.txt and llms.txt in agreement

• Write the policy down. A short note of which AI agents are allowed and why makes future edits a check against a decision rather than a guess.
• Change robots.txt in one place. If a plugin, a CDN setting and a physical file can all affect it, pick one and switch the others off.
• Test on deploy. Run the agent-by-URL script against the live site after every deployment that touches robots.txt, llms.txt or URL structure.
• Review after platform changes. CDN features, SEO plugin updates and host migrations can all add or reset rules.
• Remember what robots.txt cannot do. It is a request that well behaved crawlers follow. User-requested fetchers from some providers may not apply it, and crawlers that ignore it entirely are only stopped at the firewall.

For firewall and CDN refusals, which produce error codes rather than polite non-visits, see the AI crawler access guide. For robots.txt mistakes that affect ordinary search crawlers, see accidental robots.txt blocks. Nothing here is legal advice about how AI companies may use your content.