AI Agent Compatibility Example: Robots, Schema and Form Issues
This example shows the aiwebpageseo AI Agent Compatibility Audit for a SaaS marketing site. Agent Score is 58 out of 100. The audit found 5 of 8 AI crawler directives configured, 62% structured data coverage, 23 form accessibility issues blocking autonomous agents, and no llms.txt file. ChatGPT Operator and Claude Computer Use would struggle to complete signup or contact tasks on this site.
| Crawler | Purpose | Status | Recommendation |
|---|---|---|---|
| GPTBot | OpenAI training | ✅ Allowed | Keep allowed for OpenAI visibility |
| ChatGPT-User | Live answer fetch | ✅ Allowed | Critical for ChatGPT citation |
| OAI-SearchBot | OpenAI search index | 🔴 Not specified | Add User-agent: OAI-SearchBot |
| ClaudeBot | Anthropic | 🔴 Not specified | Add ClaudeBot directive |
| PerplexityBot | Perplexity | 🔴 Not specified | Add PerplexityBot directive |
| Google-Extended | Gemini training | ✅ Allowed | Required for Gemini visibility |
| CCBot | Common Crawl | ⚠ Default deny | Decide deliberately — CC feeds many AI models |
| Bytespider | ByteDance | ⚠ Default deny | Block if China traffic not relevant |
Signup form — missing autocomplete
Your signup form has no autocomplete attributes on email, password or name fields. ChatGPT Operator and Claude Computer Use rely on autocomplete tokens to fill fields correctly. Add autocomplete="email", autocomplete="new-password", autocomplete="given-name".
Checkout — fields missing labels
Six checkout fields use placeholder text but no <label> elements. Agents cannot reliably identify what each field expects. Add explicit <label for="..."> tags for every input.
Captcha on contact form
hCaptcha blocks agent-driven contact submissions. Consider Cloudflare Turnstile or invisible captcha that allows verified agent traffic through.
| Issue | Count | Example | Impact |
|---|---|---|---|
| "Click here" or "Learn more" | 14 | <a>Learn more</a> | ⚠ Agents can't predict destination |
| Buttons without text | 6 | <button><svg/></button> | 🔴 Critical — no aria-label |
| Checkout reachable in > 4 clicks | 1 path | Home → category → product → cart → form → checkout | ⚠ Reduce to 3 clicks |
| Critical action behind hover-only menu | 1 | Pricing link in hover dropdown | 🔴 Agents can't trigger hover |
An agent score of 58 out of 100 is a single number covering three things that fail independently and are fixed by different people. Splitting them is the first useful thing you can do with this report.
Can an automated client fetch the page at all?
Governed by robots.txt, the WAF, bot management and rate limiting. Binary, cheap to test, and owned by infrastructure. Five of eight directives configured means three named agents have no explicit policy, which usually means they inherit whatever the default rule does — and the default rule in most managed bot-protection products is to challenge unfamiliar clients.
Is the substance in the served HTML, and is it described correctly?
Governed by rendering strategy and structured data. 62% schema coverage means over a third of pages assert nothing machine-readable about what they are. Owned by engineering and content together.
Can a client complete a task without a human?
Governed by form semantics, control labelling and challenge design. This is where the 23 form issues and 14 generic buttons live, and it is the layer nearly every site fails hardest — because it has never been tested by anything other than a person with a mouse.
None of these is a Google ranking factor, and the report should not be read as if it were. Two of the three are accessibility and correctness work that happens to be legible to machines; the third is infrastructure hygiene. Treat the score as a checklist, not as a proxy for search performance.
The single biggest mistake in this whole field is treating "AI crawlers" as one category and writing a robots.txt that allows or blocks them all together. They do different jobs and the cost of blocking each one is different.
| Agent | What it is for | Blocking it costs you |
|---|---|---|
| GPTBot | OpenAI model training | Nothing in ChatGPT's cited answers. This is a content-licensing decision, not an SEO one. |
| OAI-SearchBot | OpenAI search and citation | Eligibility to be cited as a source in ChatGPT. |
| ChatGPT-User | User-initiated fetch | A user's ability to point ChatGPT at your page and have it read. |
| ClaudeBot | Anthropic web crawling | Reachability by Claude. |
| PerplexityBot | Perplexity indexing | Eligibility as a Perplexity source. |
| Google-Extended | Gemini training control | Nothing in Google Search and nothing in AI Overviews. |
| Googlebot | Google Search index | Google Search itself. Never block this. |
Two conclusions follow, and both contradict advice you will read elsewhere. Blocking GPTBot does not remove you from ChatGPT, because ChatGPT's citation path is served by different agents. And disallowing Google-Extended has no effect on Google Search, so a publisher can decline Gemini training without paying any Search cost at all.
There is also a limit worth stating: robots.txt is a request, honoured by the operators who choose to honour it. It is not an access control. If you need to actually prevent retrieval rather than politely ask for it, that is a WAF rule or an authentication wall, and it will apply to everything — including the crawlers you wanted.
Every form defect this audit found is already a defect under WCAG, and has been for years. A field without a programmatically associated label cannot be announced by a screen reader. An input without a type attribute or an autocomplete token cannot be filled from stored credentials. A validation error signalled only by turning a border red communicates nothing to a user who cannot see the border.
That an automated client also cannot use the form is a consequence of the same defect, not a separate problem. This matters practically, because it changes the argument you make internally. "AI agents cannot complete our signup" is speculative and easy to deprioritise. "Our signup form fails WCAG 2.2 success criteria for labels and identified input purpose" is a compliance statement with a legal dimension, and it produces the same fix.
Every input needs a programmatic label
A visible <label for="..."> bound to the input id. Placeholder text is not a label — it disappears on focus, it is often not announced, and it fails contrast requirements at typical placeholder colours. aria-label is a fallback for controls with no visible text, not a substitute for a visible one.
Use the standard tokens
autocomplete="email", "given-name", "postal-code", "cc-number". These are a fixed vocabulary in the HTML specification, and they are what tells any client — a password manager, a browser, a screen reader announcing purpose, an automated agent — what a box is for. This is the single highest-value change on the list because it is mechanical and the vocabulary is already defined.
type="email", type="tel", type="number"
Correct types give mobile users the right keyboard, give browsers free validation, and tell any parser what shape of value is expected. Defaulting everything to type="text" throws all three away for nothing.
Errors must be in text, and associated with the field
aria-describedby pointing at a message that says what is wrong and how to fix it. Colour alone is not an error message. A form that fails silently is a form that cannot be completed by anyone operating it without sight of the styling.
A captcha on the contact form is flagged because it stops automated completion. It stops automated completion because that is precisely what it was installed to do. Removing it to please an agent audit is a bad trade if the abuse it prevents is real.
Ask instead what the challenge is protecting and whether the protection is still proportionate. Many captchas are on forms that would receive negligible spam without them, added years ago by someone who has left, and never revisited. Where the risk is genuine, risk-based challenges that only escalate for suspicious signals will let ordinary traffic through untouched — and that includes both a legitimate user and a legitimate agent acting for one.
There is a third option that is better than either: give automated clients a route that does not involve driving your interface. A documented API endpoint, or an MCP server exposing the same capability, lets an agent do the task properly, with authentication, rate limiting and an audit trail — none of which you get when an agent is puppeteering a form built for a human. Accessibility work makes your UI operable. An API makes the UI unnecessary.
Fourteen controls on this site have accessible names that describe nothing — the "Learn more" and "Submit" family. It is worth being precise about the harm, because the usual justification for fixing them is wrong.
Generic anchor text does not waste link equity. Anchor text does not change how much ranking signal a link passes; it describes the destination. The real cost is identifiability. A screen-reader user who lists the controls on a page sees fourteen entries that say "Learn more" and cannot tell them apart without navigating back into the surrounding text. An automated client choosing which control to operate faces the identical problem, for the identical reason.
The fix is to write what the control does: "Start a free audit", "Download the schema report", "View pricing plans". If the visual design requires a short label, the accessible name can carry the detail via aria-label while the visible text stays short — but a visible label that already says the right thing is better than an invisible one that corrects it.
Fix the form labels, types and autocomplete tokens
Accessibility compliance and agent operability in one change. Mechanical, testable, and the largest component of the score gap.
Write the three missing robots directives deliberately
Not as a block-everything or allow-everything template. Decide training access and retrieval access separately, and document why.
Rename the 14 generic controls
Cheap copy work with an unambiguous accessibility benefit. No ranking claim attached.
Raise schema coverage above 62% — accurately
Mark up what the page actually contains and no more. Structured data must describe the visible page; inventing types the page does not support is a policy violation, and there is no verified evidence that denser markup wins citations.
Publish llms.txt
Low cost, unproven benefit, no downside. Do it last and do not expect it to move anything on its own.
What is an AI agent compatibility audit actually testing?
It tests whether an automated client that is not a browser can read your page and act on it. That splits into three independent questions. Can it reach you at all — do the named crawlers receive a 200 rather than a 403 from your robots.txt, your WAF and your rate limiter? Can it understand what it read — is the substance present in the served HTML, and is the structured data valid and consistent with the visible text? And can it complete a task — are your forms labelled, are your inputs typed and autocompletable, and do your buttons say what they do? A page can pass the first two and fail the third completely, which is what an agent score of 58 usually means.
Does a low agent score hurt my Google rankings?
Not directly, and it is important to say so plainly. There is no agent-compatibility ranking factor in Google Search. Most of what this audit measures is either invisible to Google's ranking systems or overlaps with things that matter for other reasons. The overlap is where the value sits: server-side rendering of important content helps every crawler, form labels and typed inputs are accessibility requirements under WCAG that also happen to make a form machine-operable, and descriptive link text is better for screen-reader users before it is better for anything else. Fix these because they are correct, not because you expect a ranking movement.
Why do 23 form issues matter more than the missing llms.txt?
Because they are the difference between a page an agent can read and a page an agent can use. An llms.txt file is a curated Markdown map of your site; adoption by assistant operators is partial and its effect is unproven, so it is cheap hygiene rather than a lever. A form with unlabelled fields and no autocomplete attributes is a hard stop: an automated client cannot reliably determine which box takes the email address, so any task that ends in a submission fails. If you are ordering the work by consequence rather than by novelty, the forms come first.
Should I remove the captcha so agents can complete forms?
Not as a blanket decision. A captcha exists to stop abuse and it usually works, so removing it to accommodate agents trades a real security control for a speculative benefit. The better framing is to ask what the captcha is protecting and whether the protection is proportionate. A captcha on a contact form that receives four submissions a day is often protecting nothing; a captcha on account signup is protecting a great deal. Where you do want automated clients to succeed, the modern options are risk-based challenges that only escalate for suspicious traffic, or a documented API endpoint that gives agents a legitimate route that does not involve driving your UI at all.
What makes a button 'generic', and does it really matter?
A generic button is one whose accessible name does not describe its action — "Learn more", "Click here", "Submit", "Go". It matters, but not for the reason usually given: anchor and button text does not change how much ranking signal passes anywhere. What it changes is whether the destination or the action can be identified without the surrounding visual context. That is a hard requirement for a screen-reader user pulling up a list of the page's controls, and it is the same requirement for an automated client choosing which control to operate. The accessibility argument is the stronger one and it is the one worth making internally, because it is the one that is actually enforceable.
Is blocking GPTBot the same as blocking ChatGPT?
No, and conflating them is the most common error in this whole area. GPTBot fetches content for OpenAI model training. ChatGPT-User handles user-initiated fetches, and OAI-SearchBot serves the search-and-cite path that produces linked sources in an answer. Disallowing GPTBot removes you from a training corpus and has no effect on whether ChatGPT can cite you. Similarly, Google-Extended governs Gemini training only and has no bearing on Google Search or on AI Overviews, which are served through ordinary Googlebot access. Decide training and retrieval separately, because they are separate decisions with separate costs.
Does server-side rendering matter if Googlebot can execute JavaScript?
Googlebot does render JavaScript, but it is the exception rather than the rule among automated clients. Most retrieval systems fetch the HTML and parse it; they do not run a browser engine, they do not wait for hydration, and they do not click anything. Content that only exists after a client-side fetch resolves is, to those clients, content that does not exist. This is why the served-HTML test is on the audit at all: it is not a claim that rendering is impossible, it is a claim that depending on it narrows the set of things that can read you, and the narrowing is invisible until you look at a log.
Related Demo Reports
Run AI Agent Compatibility Audit on Your Own Site
Get your real agent compatibility audit covering all 8 categories — robots, schema, semantic HTML, forms, actions, content, auth and llms.txt.