AI Agent Compatibility: Letting ChatGPT and Claude Use Your Site
Learn how AI agents like ChatGPT Operator and Claude Computer Use interact with websites, and how to make your forms, buttons and navigation agent-friendly.
What you will learn in this guide
- What AI agents are and how they differ from search crawlers
- Why generic crawler signals don't apply to agents
- How to make forms agent-fillable
- Why "Click here" buttons fail for agents
- How llms.txt and accessibility overlap with agent compatibility
1 What is an AI agent?
An AI agent is an LLM-powered system that browses websites on behalf of a user — clicking links, filling forms, and completing tasks. Examples: ChatGPT Operator, Claude Computer Use, Anthropic Computer-Using Agent, Perplexity Auto.
2 How agents differ from crawlers
| Crawler | Agent |
|---|---|
| Reads HTML once | Interacts continuously |
| Doesn't click buttons | Clicks every button on the path to a task |
| Ignores forms | Must fill forms to complete tasks |
| Stateless | Maintains session and context |
| Cares about robots.txt for indexing | Cares about robots.txt for permission to act |
3 How to make your site agent-friendly
- 1Add autocomplete attributes to every form fieldAgents use autocomplete tokens to know what each field expects.
autocomplete="email",autocomplete="name",autocomplete="tel"etc. - 2Use real
<label>elementsNot just placeholder text.<label for="email">Email</label>with matching id on the input. Agents read labels to understand fields. - 3Replace "Click here" with descriptive button textBad: "Click here". Good: "Add to basket", "Submit booking", "Continue to checkout". Agents predict what a button does based on its text.
- 4Ensure every key action is reachable by keyboardTest with Tab and Enter. If you can't complete a task with just the keyboard, an agent often can't either.
4 AI crawler robots.txt directives
Decide which AI agents you allow. Add to your robots.txt:
5 Before anything else: check your logs
Every recommendation in this guide is worthless if agents cannot reach your origin, and there is exactly one place where that fact is recorded: your access log.
Filter it for the agent user-agents — ChatGPT-User, ClaudeBot, PerplexityBot, OAI-SearchBot, Googlebot — and look at two things. Are they arriving at all? And what status code are they getting?
The answer is frequently a surprise. Sometimes the traffic is negligible, and the honest conclusion is that this whole topic is premature for your site and your effort belongs elsewhere. Far more often, the agents are arriving and being served 403s by a bot-mitigation rule that somebody enabled by default years ago and nobody has looked at since.
6 Training and retrieval are different decisions
The robots.txt block above allows ChatGPT-User, ClaudeBot, PerplexityBot and OAI-SearchBot. It is worth understanding what each of those directives actually controls, because the agents are not interchangeable and the most common configuration mistake follows from treating them as though they were.
| Agent | What it does | Blocking it means |
|---|---|---|
| GPTBot | OpenAI training crawler | Your content is not used to train models. It does not remove you from ChatGPT's answers. |
| OAI-SearchBot | Fetches pages for ChatGPT's search | You are not retrieved as a source for live answers. This is the one that costs you citations. |
| ChatGPT-User | Fetches a page a user asked about | ChatGPT cannot open your page when a user asks it to. |
| Google-Extended | Gemini training only | No effect on Google Search, on rankings, or on AI Overviews. |
The pattern to avoid: a site blocks GPTBot to opt out of training, feels it has taken a position, and is then surprised to still be quoted by ChatGPT. Or the reverse — a site blocks everything with "GPT" or "AI" in the name and quietly removes itself from every assistant's citations, having intended only to decline training.
7 Put the substance in the served HTML
Agents divide into two kinds and the distinction decides how much of your site they can see. Browser-driving agents genuinely run a browser: they wait for scripts, they see the rendered page, they can click. Retrieval-oriented fetchers generally take the HTML as it arrives from your server and parse it — no rendering, no waiting, no JavaScript.
For that second group, anything your page assembles client-side after load does not exist. Not "ranks lower" — does not exist. Prices fetched from an API on mount. Availability rendered by a framework after hydration. FAQ answers inside an accordion that only populates on click. Product specifications in a tab that loads its content when the tab is selected.
You can check this in ten seconds without any tooling: view the page source, or fetch the URL with curl, and search for the text you care about. If your price is not in that response, the fetchers cannot see your price. It does not matter how perfect it looks in a browser.
The fix is server-side rendering, or static generation, or simply putting the important content in the initial HTML and enhancing it afterwards. It costs nothing, it helps search engines and assistants alike, and it makes the question of what each agent supports irrelevant.
8 Agent compatibility is accessibility with a new name
Read back through the practical advice in this guide — real label elements, autocomplete tokens, buttons that are buttons, meaningful link text, content present in the markup rather than conjured by script — and notice what it amounts to. It is the accessibility checklist. Almost line for line.
That is not a coincidence. A screen reader and an agent face the same problem: they must understand a page without seeing it. Both need the structure to carry the meaning, because neither can infer from a layout that a bold blue rectangle is the submit button, or that the text floating above a box is its label.
- 1A placeholder is not a labelIt disappears on focus, it fails contrast requirements, and it gives a machine nothing to bind the field to. Use a real <label for=""> with a matching id.
- 2A div with a click handler is not a buttonIt cannot be focused, cannot be activated by keyboard, and announces nothing. Use <button>. Every agent, every screen reader and every keyboard user benefits immediately.
- 3"Click here" is not link textLink text is a description of the destination. It is what a machine uses to understand what it is about to open, and what a screen-reader user hears in a list of links stripped of context.
- 4Errors must be in the markupA validation failure shown only as a fading toast is invisible to anything that is not watching the screen. Render it in the DOM, associated with the field it concerns.
The encouraging consequence: this work was already worth doing, it is legally required in many jurisdictions, and it has a constituency of real human beings who benefit from it today. Agent compatibility is a second reason to do something you should be doing anyway — not a new discipline requiring new tooling, and certainly not a special interface built alongside your real one.
9 Frequently asked questions
Check rather than assume, because the answer varies enormously by site and the only place it is recorded is your own access log. Filter for the agent user-agents — ChatGPT-User, ClaudeBot, PerplexityBot, OAI-SearchBot and the rest — and look at both the volume and the status codes they receive. What you find is frequently a surprise in one of two directions: either the traffic is negligible and the whole topic is premature for your site, or the agents are being served 403s by a bot-mitigation rule nobody remembers configuring. Neither of those is visible from anywhere except the log.
This is the single most common cause of an invisible site, and it produces no error anybody notices. Bot-mitigation rules — at Cloudflare, at your CDN, in a security plugin — frequently match on unfamiliar user-agent strings or on request patterns that look automated, and they were usually enabled by default rather than by decision. The result is a permissive robots.txt sitting in front of an origin that silently refuses every agent request. robots.txt is a request to well-behaved clients; the WAF is what actually decides who gets a response. Only the access log tells you which one is winning.
Some do and some do not, and building on the assumption that they all do is the expensive mistake. Browser-driving agents genuinely operate a browser, so they see what a user sees after scripts run. Retrieval-oriented fetchers generally take the HTML as served and parse it, which means anything assembled client-side after load simply does not exist for them — your prices, your availability, your FAQ answers, your product specifications. The safe rule is to put the substance in the served HTML. It costs nothing, it helps every client including search engines, and it removes the question entirely.
The same things a screen reader needs, which is the useful insight here: agent compatibility and accessibility are very nearly the same discipline. A real label element associated with its input, so the field has a name rather than a placeholder that vanishes on focus. An autocomplete token, so the agent knows this field wants an email and that one a postcode. A submit button that is a button, not a div with a click handler. Error messages rendered in the markup rather than only in a toast that disappears. A form that a blind user can complete is a form an agent can complete.
No — and the sites that have tried have mostly ended up maintaining two things and getting neither right. There is no agent-specific protocol you need to adopt, no schema type that admits you, and no file that grants you agent compatibility. What agents need is a site whose structure honestly reflects its content: semantic HTML, real form controls, meaningful link text, substance in the served markup, and accurate structured data where it disambiguates something. That is the same list you would write for accessibility, and doing it once serves everybody.
No, and this confusion costs sites real visibility. GPTBot is OpenAI's training crawler; blocking it removes your content from a training corpus. The agents that fetch pages to answer a live user's question are different — OAI-SearchBot and ChatGPT-User — and they are governed by their own directives. A site that blocks GPTBot to opt out of training, believing it has done nothing else, has done nothing else. A site that blocks all three has removed itself from ChatGPT's cited answers, and usually did not intend to. Decide about training and about retrieval separately, because they are separate decisions.
Google-Extended controls whether your content is used for Gemini model training. It has no effect whatsoever on Google Search, on your rankings, or on whether you appear in AI Overviews — those run on ordinary Googlebot access and ordinary Search eligibility. Sites disallow Google-Extended believing they are opting out of AI Overviews and are then puzzled to find themselves still appearing in them; sites also avoid disallowing it for fear of harming their rankings, which is equally unfounded. It does one narrow thing, and that thing is training.