AI Agent Readiness: Complete Discovery Setup Guide

The web is changing. Alongside human visitors and search engine crawlers, a new category of visitor is emerging — AI agents. These are automated systems that browse, read, analyse and interact with websites on behalf of users. This guide covers everything you need to know to make your site fully discoverable and accessible to AI agents.

What is AI Agent Readiness?

AI Agent Readiness is the practice of publishing the right signals, files and headers that AI agents need to discover and interact with your site. It covers four areas:

Discovery Files

1. llms.txt

The most important file. Place it at /llms.txt at your domain root. It tells AI language models what your site is about, lists your most important pages, and declares how you want your content used.

# Your Site Name
> One sentence description of what your site does.

## Key pages
- [Page Title](https://example.com/page): Brief description.

## About
Two or three paragraphs about your site, mission and content.

## AI usage permissions
AI models are permitted to index, summarise and cite content from this site.

2. API Catalog (RFC 9727)

Create /.well-known/api-catalog returning application/linkset+json. This tells AI agents what APIs your site exposes.

{
  "linkset": [{
    "anchor": "https://example.com",
    "service-desc": [{"href": "https://example.com/llms.txt", "type": "text/plain"}],
    "service-doc": [{"href": "https://example.com/docs", "type": "text/html"}]
  }]
}

3. MCP Server Card

Create /.well-known/mcp/server-card.json. This enables AI assistants using the Model Context Protocol to discover your tools and capabilities.

{
  "serverInfo": {"name": "Your Site", "version": "1.0.0", "description": "What your site does"},
  "transport": {"type": "http", "endpoint": "https://example.com/api/"},
  "capabilities": {"tools": true, "resources": true},
  "documentation": "https://example.com/docs"
}

4. A2A Agent Card

Create /.well-known/agent-card.json. This enables agent-to-agent discovery and interaction.

{
  "name": "Your Site",
  "version": "1.0.0",
  "description": "What your agent does",
  "url": "https://example.com",
  "skills": [
    {"id": "main-skill", "name": "Skill Name", "description": "What this skill does"}
  ]
}

5. Agent Skills Index

Create /.well-known/agent-skills/index.json listing your site's capabilities per the Agent Skills Discovery RFC v0.2.0.

{
  "$schema": "https://agentskills.io/schema/v0.2.0/index.json",
  "skills": [
    {"name": "Skill Name", "type": "skill-md", "description": "What it does", "url": "https://example.com/guide"}
  ]
}

Content Access

llms.md

Create /llms.md — a markdown-formatted version of your llms.txt. Some agents prefer markdown over plain text.

Markdown Negotiation

Configure your server to return markdown content when an agent sends Accept: text/markdown. In NGINX:

location = / {
  if ($http_accept ~* "text/markdown") {
    rewrite ^ /llms.md last;
  }
  try_files /index.html @fallback;
}
location ~* \.md$ {
  default_type text/markdown;
  add_header Content-Type "text/markdown; charset=utf-8" always;
}

Content-Signal in robots.txt

Add this directive to your robots.txt to declare your AI content preferences:

Content-Signal: ai-train=no, search=yes, ai-input=no

Security Headers

Three headers matter most for agent readiness:

AI Crawler Permissions

Explicitly allow the major AI crawlers in your robots.txt:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: anthropic-ai
Allow: /

Frequently Asked Questions