How to Fix Missing or Broken llms.txt

llms.txt is a proposed standard (llmstxt.org) — a markdown file at /llms.txt that tells AI agents where your best, most useful content lives. Think of it as a curated AI-sitemap. While XML sitemaps list every indexable URL, llms.txt highlights only the content you want LLMs to surface. The cost is low (one file). The upside grows as more AI engines adopt the standard.

1. The format

llms.txt is markdown. Structure:

# Site Name

> One-paragraph blockquote describing what your site does. AI parses
> this as the authoritative description of your site.

Optional plain text introduction paragraph. Sets context for the rest.

## Section 1 (e.g. Documentation)

- [Page Title](https://example.com/docs/page1): One-line description
- [Another Page](https://example.com/docs/page2): One-line description

## Section 2 (e.g. Guides)

- [Guide Title](https://example.com/guides/x): Description

## Optional

- [Less critical link](https://example.com/extra): Description

2. Real example

# AIWebPageSEO

> The SEO audit platform for the AI era. Site audits, schema validation,
> AI visibility tracking, and agent readiness — pay-as-you-go from 1p.

AIWebPageSEO helps teams optimise for traditional search AND for AI answer
engines (ChatGPT, Claude, Perplexity, Gemini).

## Core Audits

- [Site Audit](https://aiwebpageseo.com/seo-audit-platform.html): Full technical SEO crawler with 200+ checks
- [Schema Debugger](https://aiwebpageseo.com/schema-tools.html#debugger): JSON-LD validation against schema.org + Google rich results
- [AI Visibility Tracker](https://aiwebpageseo.com/ai-visibility.html): Track citations across ChatGPT, Claude, Perplexity
- [Agent Readiness](https://aiwebpageseo.com/agent-readiness.html): How crawlable, parseable, AI-friendly is your site

## Guides

- [Site Audit Guide](https://aiwebpageseo.com/aipageseo-demo-pages/site-audit-guide.html): How site audits work
- [AEO Guide](https://aiwebpageseo.com/aipageseo-demo-pages/aeo-guide.html): Answer Engine Optimisation explained
- [Schema Guide](https://aiwebpageseo.com/aipageseo-demo-pages/schema-debugger-guide.html): JSON-LD structured data

## Pricing

- [PAYG Pricing](https://aiwebpageseo.com/pricing.html): From 1p per audit, no subscription required

## Optional

- [Blog](https://aiwebpageseo.com/blog): Industry updates and how-tos
- [Community](https://aiwebpageseo.com/community): Forum for users

3. Create llms.txt

4. nginx / Apache config

nginx

location = /llms.txt {
  default_type "text/markdown; charset=utf-8";
  add_header Content-Type "text/markdown; charset=utf-8" always;
}

Apache (.htaccess)

<Files "llms.txt">
  ForceType "text/markdown; charset=utf-8"
</Files>

5. llms-full.txt (optional, recommended)

The full-content variant — concatenates actual content of linked pages into one markdown file. Lets an AI agent fetch your entire useful content in one request.

# AIWebPageSEO — Full Content

## Site Audit

[Full content of /seo-audit-platform.html in markdown]

---

## Schema Debugger

[Full content of /schema-tools.html#debugger in markdown]

---

## AEO Guide

[Full content of /aipageseo-demo-pages/aeo-guide.html in markdown]

Generate via build script:

// scripts/generate-llms-full.js
const fs = require('fs');
const TurndownService = require('turndown');
const td = new TurndownService();

const pages = [
  { title: 'Site Audit', file: 'public/seo-audit-platform.html' },
  { title: 'AEO Guide', file: 'public/aipageseo-demo-pages/aeo-guide.html' },
  // ...
];

let out = `# Site Name — Full Content\n\n`;
for (const p of pages) {
  const html = fs.readFileSync(p.file, 'utf-8');
  // Extract main content (skip nav/footer)
  const main = extractMain(html);
  out += `## ${p.title}\n\n${td.turndown(main)}\n\n---\n\n`;
}

fs.writeFileSync('public/llms-full.txt', out);

6. Reference from robots.txt (optional)

# robots.txt — comment for AI agent discoverability
# llms.txt: https://example.com/llms.txt
# llms-full.txt: https://example.com/llms-full.txt

User-agent: *
Allow: /

Not a standard directive but increasingly recognised by AI tooling. Free signal.

7. Maintain

llms.txt should stay current. When you publish important new content, add it. When pages are deprecated, remove entries. Consider regenerating on every deploy:

// package.json
"scripts": {
  "build": "next build && node scripts/generate-llms.js",
  "predeploy": "npm run build"
}

8. Verify the fix

curl -I https://example.com/llms.txt
# Should return 200 OK with content-type: text/markdown

curl https://example.com/llms.txt | head -30
# Should show your markdown structure