Agent Readiness Fixes: Make Your Content Readable to AI Crawlers
AI crawlers such as GPTBot, ClaudeBot and PerplexityBot visit websites to collect text that AI assistants later draw on when they answer questions. They are less patient than Googlebot: most do not run JavaScript, they can be stopped by robots.txt or a firewall, and they struggle with pages where the real content is buried in scripts, images or div soup.
The Agent Readiness Audit requests your page with those crawlers' user agents and reports what stops them reading it. This page explains each finding, the order to tackle them in and the guide that fixes each one.
Five gates between an AI crawler and your content
It helps to picture a crawler's visit as a series of gates. Each one can stop the visit, and a failure at an early gate makes everything after it irrelevant. The audit's findings map onto these gates.
- Permission. The crawler reads
/robots.txtfirst. If its user agent, or*, is disallowed, a well-behaved crawler leaves without fetching the page. - Delivery. The request then passes through your host, CDN and firewall. Bot protection can answer with a challenge page, a 403 error or a timeout, even though robots.txt said yes. The crawler records nothing useful.
- Rendering. The crawler receives the HTML your server sends. Most AI crawlers stop here and do not run scripts, so anything that only appears after JavaScript runs is missing from their copy.
- Extraction. From that HTML, the crawler has to separate the main content from menus, cookie banners, sidebars and footers. Clear semantic elements make this easy; hundreds of nested divs make it guesswork.
- Understanding. Finally, the text is interpreted. Structured data, a clear author, a single canonical URL and consistent facts help the system decide what the page is about, who stands behind it and which version to trust.
Work through the gates in that order. There is no point polishing JSON-LD on a page that the firewall is quietly refusing to serve.
The fix guides
Each guide takes one audit finding and covers why it happens, how to fix it, how to check the fix worked and the questions site owners ask about it.
Access: can the crawler get in?
Fixing AI crawler robots.txt blocks explains how to read your robots.txt the way a crawler does, why a blanket rule added years ago may now be excluding AI search tools, and how to allow the crawlers that power AI answers while making a separate choice about crawlers that collect training data. Adding an llms.txt file covers the optional Markdown file that points language models at your most useful pages, what to put in it and how to keep it short enough to be useful.
Rendering and extraction: can it read what it fetched?
Fixing slow rendering for AI crawlers
covers server-side rendering, static generation and pre-rendering, and how to confirm that bots receive
finished HTML rather than an empty shell. Fixing
poor content extractability deals with facts locked inside images, tables built from divs, text loaded
into tabs and accordions by script, and the reader-mode test as a quick check. Fixing
non-semantic HTML shows how to wrap the real content in <main> and <article>,
use headings in order and mark navigation as <nav>, usually without changing how the
page looks.
Understanding: can it tell what the page means and who wrote it?
Fixing missing or incomplete JSON-LD
covers which schema.org types suit which page types and how to connect them with @graph.
Fixing missing author and trust
signals covers bylines, author pages, about and contact pages and cited sources. Fixing
canonical and URL conflicts covers pages reachable at several addresses, parameter clutter and redirect
chains that split one page into many copies.
By platform
The platform guides show where these problems usually start and which settings fix them: WordPress, where themes, SEO plugins and security plugins all touch crawler access; Shopify, where robots.txt is edited through a theme template and product data needs to be complete; and React and Next.js, where the choice between client rendering, server rendering and static generation decides what a crawler sees.
See your page the way a crawler sees it
You do not need special software to get a rough idea of what an AI crawler receives. Three quick checks, done before and after each fix, catch most problems the audit reports.
1. Fetch the raw HTML with a crawler's user agent
From a terminal, request the page while identifying as a crawler, then search the result for a sentence you know is on the page:
curl -A "GPTBot" -s https://www.example.co.uk/services/ | grep -i "free quote"If the sentence is missing, either the text is added by JavaScript or something along the way served
a different response. Try again with -I to see the status code: a 403, 429 or 503 points
to a firewall or rate limit rather than your content. Remember that a user agent string can be faked,
so some firewalls also check the crawler's published IP ranges; a curl test from your own machine will
not reproduce that part.
2. Turn JavaScript off
Disable JavaScript in your browser's developer tools and reload. What remains is close to what a non-rendering crawler gets. Blank sections, "Loading..." placeholders and missing prices are the usual findings.
3. Use reader mode
Firefox Reader View and similar features try to pick out the main article. If reader mode shows the menu, the cookie notice and nothing else, an extraction system is likely to struggle in the same way.
Then check your logs
Search your server or CDN access logs for the crawler names. Seeing GPTBot or ClaudeBot
with 200 responses shows they are getting through. Seeing them only with 403 responses, or not at all,
points to a block somewhere upstream of your pages.
Deciding which AI crawlers to allow
Whether AI crawlers should read your site is a business decision, not a technical fault. The audit reports what is blocked; it does not assume you want everything open. What matters is that the result is deliberate.
Most AI companies now run separate crawlers for separate jobs. Broadly there are three kinds:
| Kind of crawler | What it does with your page | Effect of blocking it |
|---|---|---|
| Training collection | Gathers text that may be used to train future models | Your content is less likely to be part of future training data. Answers from AI search tools are not directly affected. |
| Search indexing | Builds the index an AI assistant searches when it needs current sources | Your pages are unlikely to be found, linked or cited in that assistant's answers. |
| User-requested fetch | Fetches one page because a person asked the assistant about it or pasted its link | The assistant cannot read your page even when someone asks it to. |
Because these are separate user agents, you can say no to training and yes to search in the same robots.txt file. Each provider documents its own names, and new ones appear, so check the provider's current documentation rather than copying an old list. The crawler access guide has worked examples.
Robots.txt is a request, not a lock. The major providers say their crawlers respect it, but a rule does nothing against a scraper that chooses to ignore it. If you need to enforce a block, it has to happen at the firewall or CDN.
Which findings to fix first
Findings do not all carry the same weight. Fix them in this order, because each step depends on the one before.
- Unintended blocks. A robots.txt rule or firewall setting that stops crawlers you actually want is the single biggest problem the audit can find. It takes minutes to fix and nothing else helps until it is fixed.
- Empty HTML. If the server sends a shell and JavaScript builds the page, most AI crawlers see almost nothing. This is the most expensive fix, often needing a rendering change, but it unlocks every page built the same way.
- Key facts outside the text. Prices, opening hours, specifications and answers that live only in images, PDFs or script-loaded tabs. Move them into plain HTML text.
- Duplicate URLs. When one page answers at several addresses, crawlers split their attention between copies. Pick one URL and point everything else at it.
- Structure and structured data. Semantic elements and JSON-LD make extraction more accurate once the content is there to extract.
- Authorship and trust. Named authors, an about page and cited sources help AI systems judge whether a page is a reliable source.
- llms.txt. Worth adding, but it only helps once everything above works, and support for it among AI providers is still limited.
A sensible order of work
- Run the Agent Readiness Audit on your home page and on one page from each template: a product, an article, a service page, a category page.
- Open your live robots.txt and read every rule that names an AI crawler or applies to all user agents. Decide deliberately which crawlers you want.
- Check your CDN, host and security plugin for AI-bot or bot-fight settings, and confirm the crawlers you allow can actually get a 200 response.
- Run the curl and JavaScript-off tests on each template. Where text is missing, plan a move to server rendering, static generation or pre-rendering.
- Move facts out of images and script-only widgets into HTML text, and replace div-built tables with
real
<table>markup. - Wrap the main content in
<main>or<article>, fix heading order and set one canonical URL per page. - Add or complete JSON-LD for each template and link the author, organisation and page together.
- Add bylines and author pages where content is written by a person, then add an llms.txt file if you want one.
- Re-run the audit and repeat after theme changes, plugin updates and platform migrations, which are when crawler access most often breaks.
For background on each area, see the Agent Readiness Guide. The beginner tutorial walks through a first audit step by step, the example report shows what the findings look like, and the llms.txt Guide goes deeper on that one file.
1. Is agent readiness the same as SEO?
It overlaps a lot, but it is not the same. Google renders JavaScript, tolerates messy HTML and has decades of signals to fall back on. Most AI crawlers do far less: they fetch the raw HTML, often skip scripts, and extract what they can from what arrives. A page that ranks well in Google can still be close to invisible to an AI crawler, which is why the audit checks the things Google tends to forgive.
2. Does blocking GPTBot remove me from ChatGPT?
Not on its own. OpenAI documents GPTBot as its training crawler, while search results in ChatGPT rely on a separate crawler and user-requested fetches use another name again. Blocking GPTBot therefore mainly affects training data rather than whether ChatGPT search can find and link to you. Other providers split their crawlers in a similar way. Check each provider's current documentation, because the names and their stated purposes do change from time to time.
3. Do AI crawlers run JavaScript?
Most AI crawlers fetch the HTML your server sends and do not execute scripts, or execute them only in limited ways. Google's crawlers are the main exception because they share Google's rendering system. The safe assumption is that anything added to the page by JavaScript after it loads may not be seen at all. Server-side rendering, static generation or pre-rendering make sure the full text is present in the first response every crawler receives.
4. Do I need an llms.txt file to be ready for AI?
No. It is an optional proposal, and as far as we know no major AI search provider has publicly confirmed that it uses llms.txt when choosing which sources to cite. It is cheap to add and can help tools that do read it, such as some coding assistants working with documentation. Treat it as a small extra once crawler access, rendering and page structure are right, not as a fix for any of them.
5. Will structured data make AI tools cite my site?
Nothing guarantees a citation. Structured data states clearly what a page is, who wrote it, what a product costs or when an event starts, which reduces the chance of a system misreading the page. AI providers say little about exactly how they use it, so treat it as a clarity aid rather than a ranking switch. It must match the visible text, because contradictions between markup and page undermine trust in both.
6. My robots.txt allows AI crawlers, so why does the audit say they are blocked?
Robots.txt is only the first gate. A firewall, CDN bot setting, security plugin or rate limit can still refuse the request with a 403 error, a challenge page or a timeout. Some CDNs offer a single switch to block AI crawlers, and it may be enabled without anyone on your team remembering it. Check the CDN and security settings, then look in your access logs for the crawler's name and the response code it received.
7. How often should I re-run the Agent Readiness Audit?
Run it after anything that changes how pages are built or served: a new theme, a platform migration, a new CDN or security plugin, or a switch to a JavaScript framework. Those are the moments crawler access usually breaks, often without any visible change for human visitors. Outside of that, a check every few months is sensible, because AI providers introduce new crawler names and your robots.txt may need updating to cover them.
The Agent Readiness Audit requests your page with real AI crawler user agents, checks robots.txt and llms.txt, reviews JSON-LD coverage and semantic HTML, tests how much depends on JavaScript and measures how easily the content can be extracted.
Run the Agent Readiness Audit →