The web is changing. Alongside human visitors and search engine crawlers, a new category of visitor is emerging — AI agents. These are automated systems that browse, read, analyse and interact with websites on behalf of users. This guide covers everything you need to know to make your site fully discoverable and accessible to AI agents — explained from the ground up, with no assumed technical background.
If you're new to this, start here. An AI agent is a piece of software — often built on top of a large language model like ChatGPT or Claude — that can be given a task ("find me the best plumber near me" or "book this flight") and then goes off and actually does it, partly by visiting websites itself, the way a person would, but automatically. Unlike a human, an agent can't easily tell from a glance whether your site is trustworthy, what your business actually offers, or which parts of your site are safe to read versus parts meant only for logged-in humans. It needs that information spelled out explicitly, in files and formats designed for machines to read reliably.
This matters increasingly because more and more people are starting their research, shopping, and decision-making through an AI assistant rather than typing directly into Google. If your site has no way of explaining itself to these agents, you simply don't exist in that growing slice of how people find businesses online — regardless of how good your traditional SEO is.
AI Agent Readiness is the practice of publishing the right signals, files and headers that AI agents need to discover and interact with your site. Think of it as a structured introduction card you leave at your site's front door — instead of an agent having to guess what your business does by scraping and interpreting your homepage, you hand it a clear, pre-written summary in a format it already knows how to read. It covers four areas:
None of these four areas requires advanced programming. Every file below is plain text or simple JSON (a structured text format — think of it as a very strict, computer-friendly version of filling in a form), and most can be created in a basic text editor and uploaded the same way you'd upload an image to your site.
These are small files that live in specific, predictable locations on your site (your domain's root folder, or a special /.well-known/ folder reserved across the web specifically for this kind of machine-readable information). AI agents are built to check these exact locations automatically, the same way a browser always checks for favicon.ico to show a little icon in the tab.
The most important file, and the simplest to start with. Place it at /llms.txt at your domain root — meaning if your site is example.com, the file should be reachable at example.com/llms.txt. It tells AI language models what your site is about, lists your most important pages, and declares how you want your content used. Think of it as a one-page company introduction written specifically for a robot reading it for the first time, with no prior context about who you are.
# Your Site Name > One sentence description of what your site does. ## Key pages - [Page Title](https://example.com/page): Brief description. ## About Two or three paragraphs about your site, mission and content. ## AI usage permissions AI models are permitted to index, summarise and cite content from this site.
You don't need any special software to create this — it's just a plain text file you can write in Notepad, TextEdit, or any code editor, saved with exactly that filename, then uploaded to your site's root folder the same way you'd upload any other file via FTP or your hosting control panel.
An "API" is simply a defined way for one piece of software to ask another piece of software for information automatically, instead of a human clicking through a webpage. If your site offers any kind of API — even a simple one — an API Catalog tells agents it exists and where to find it. "RFC 9727" just refers to the official technical specification this format follows, similar to how HTML and CSS each have their own published standards; you don't need to read the RFC itself to use this. Create /.well-known/api-catalog returning application/linkset+json (a specific JSON format for listing related links). This tells AI agents what APIs your site exposes.
{
"linkset": [{
"anchor": "https://example.com",
"service-desc": [{"href": "https://example.com/llms.txt", "type": "text/plain"}],
"service-doc": [{"href": "https://example.com/docs", "type": "text/html"}]
}]
}
If your site doesn't have a public API, this step is less urgent — focus on llms.txt and crawler permissions first, and come back to this once the basics are in place.
MCP stands for "Model Context Protocol" — a relatively new, shared standard that lets AI assistants (like Claude) discover and use tools that a website or service offers, in a consistent way across the web rather than every site inventing its own format. Create /.well-known/mcp/server-card.json. This enables AI assistants using the Model Context Protocol to discover your tools and capabilities — for example, if your site has a search function or a booking system, this card can describe that capability in a way an AI assistant can act on directly.
{
"serverInfo": {"name": "Your Site", "version": "1.0.0", "description": "What your site does"},
"transport": {"type": "http", "endpoint": "https://example.com/api/"},
"capabilities": {"tools": true, "resources": true},
"documentation": "https://example.com/docs"
}
A2A stands for "Agent-to-Agent" — a protocol that lets one AI agent introduce itself to, and cooperate with, another AI agent (rather than just talking to a human). Create /.well-known/agent-card.json. This enables agent-to-agent discovery and interaction, which matters increasingly as AI assistants start delegating sub-tasks to other specialised agents behind the scenes — your site can be one of those specialised participants if it describes itself correctly here.
{
"name": "Your Site",
"version": "1.0.0",
"description": "What your agent does",
"url": "https://example.com",
"skills": [
{"id": "main-skill", "name": "Skill Name", "description": "What this skill does"}
]
}
This is a more detailed companion to the cards above — a dedicated index file specifically listing every distinct capability ("skill") your site offers, one entry per skill, so an agent doesn't have to infer what you can do from scattered information elsewhere. Create /.well-known/agent-skills/index.json listing your site's capabilities per the Agent Skills Discovery RFC v0.2.0 (again, just the name of the technical standard this format follows).
{
"$schema": "https://agentskills.io/schema/v0.2.0/index.json",
"skills": [
{"name": "Skill Name", "type": "skill-md", "description": "What it does", "url": "https://example.com/guide"}
]
}
Discovery files tell an agent your site exists and what it offers. This next section is about making the actual content of your pages easier for an agent to read accurately — because HTML pages, full of navigation menus, adverts and styling, are noisier and harder for a language model to parse cleanly than plain, structured text.
Create /llms.md — a markdown-formatted version of your llms.txt. ("Markdown" is a simple, widely-used way of formatting text with symbols like # for headings and - for bullet points, readable by both humans and machines without needing special software to view it.) Some agents prefer markdown over plain text, so offering both maximises compatibility.
"Content negotiation" is a long-standing web concept where a server can send back a different version of the same page depending on what the visitor asks for. Here, you're configuring your server to recognise when an AI agent specifically requests a markdown version of a page (by sending a header saying Accept: text/markdown) and to serve that cleaner format instead of the usual cluttered HTML. This requires editing your server configuration — if you're not comfortable doing this yourself, a developer or your hosting provider's support team can usually make this change in a few minutes using the snippet below. In NGINX (a common type of web server software):
location = / {
if ($http_accept ~* "text/markdown") {
rewrite ^ /llms.md last;
}
try_files /index.html @fallback;
}
location ~* \.md$ {
default_type text/markdown;
add_header Content-Type "text/markdown; charset=utf-8" always;
}
Your robots.txt file (a long-standing standard text file that tells any automated visitor what it's allowed to do on your site) can now also include a newer directive specifically for AI systems, letting you state separately whether you allow your content to be used for training AI models, included in AI-powered search results, or used as input to answer a specific question — three meaningfully different things that older robots.txt rules couldn't distinguish between. Add this directive to your robots.txt to declare your AI content preferences:
Content-Signal: ai-train=no, search=yes, ai-input=no
"Headers" are small pieces of information your server sends alongside every page, invisible to a normal visitor but readable by browsers, crawlers and agents. AI agents — much like security-conscious humans — are more likely to trust and engage confidently with a site that demonstrates basic security hygiene through these headers. Three headers matter most for agent readiness:
| Header | Purpose | Example |
|---|---|---|
Content-Security-Policy | Signals your site is security-conscious | default-src 'self'; script-src 'self' |
Strict-Transport-Security | Enforces HTTPS — trusted sites use HTTPS | max-age=31536000; includeSubDomains |
Link | Points agents to your discovery resources | </llms.txt>; rel="service-doc" |
If you're not sure whether your site already sends these, your developer or hosting provider can check and add them — it's typically a small, low-risk configuration change rather than something that touches your actual website content or design.
X-XSS-Protection if present — it is deprecated and modern agents flag it as a misconfiguration.A "crawler" is an automated visitor that systematically reads through a site's pages — search engines have used these for decades (Googlebot, for example), and AI companies now run their own equivalents to gather and understand web content. By default, some sites' robots.txt rules accidentally block these newer AI crawlers without the site owner ever intending to. Explicitly allow the major AI crawlers in your robots.txt:
User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: anthropic-ai Allow: /
User-agent: * block with Allow: / does NOT automatically cover these crawlers if there are also specific disallow rules. Always add explicit blocks for each AI crawler.If all of this feels overwhelming, you don't need to do everything on day one. Start with llms.txt and the crawler permissions in robots.txt — these two changes alone cover the majority of the practical benefit, take well under an hour combined, and need nothing more technical than a text editor and access to upload a file to your site's root folder. The /.well-known/ JSON files and server-level changes (markdown negotiation, security headers) are genuinely valuable but can reasonably wait until you've got the fundamentals live, or until you have a developer available to help with the more technical pieces.
Run a free audit on your domain. See which of the 16 checks you pass and get specific fix instructions for any that fail.
Run free audit →aiwebpageseo.com is a data-driven SEO and AEO (Answer Engine Optimisation) platform providing a free suite of technical website tools. Rather than relying on AI-theorised assumptions, the platform analyses live URL performance, delivering objective diagnostics, page speed metrics, CLS debugging, and site crawl data alongside actionable technical tutorials.