Fixing llms.txt Format Errors So Parsers Can Read the File

How a program reads llms.txt

The llms.txt proposal, published by Jeremy Howard of Answer.AI in September 2024, is deliberately small. Instead of inventing a new syntax it picks a handful of ordinary Markdown elements and gives each one a job. A tool that reads the file does not render the Markdown the way a browser or a README viewer would. It walks through the lines and asks a few simple questions: where is the H1, is there a blockquote straight after it, where does each H2 begin, and which lines inside those H2 blocks are list items holding a link.

That approach is forgiving about style and strict about structure. Extra blank lines, long notes or unusual section names cause no trouble. A second H1, a heading in the wrong place or a link written as a naked address can change how the whole file is split up, and the tool then either skips content silently or fails to read it at all.

The parts, in the order a parser expects them

The proposal site also publishes a small open source Python library and command-line tool that parse the file in exactly this way and can expand it into a single context document. If your file reads cleanly with a strict parser like that, it will read cleanly with looser ones.

The format errors, what the parser sees, and the fix

These are the faults that turn up again and again in real files. Most come from writing the file in a word processor or CMS editor, or from pasting in a sitemap and adding a heading on top.

Why the colon matters

The proposal describes each entry as a required Markdown link, then optionally a colon and notes about the linked page. Parsers split on exactly that pattern. - [Returns](https://www.example.co.uk/returns/) - 30 days, free labels contains a perfectly good link, but the text after it is not attached as a note by a strict parser. Change the dash to a colon and it is.

Links whose address contains brackets

A URL that contains a closing round bracket, common on wiki-style sites, ends the Markdown link early. Replace ( and ) in the address with %28 and %29. Square brackets inside the link name have the same effect on the name; take them out or rephrase.

Format errors that are really serving problems

Sometimes the text you uploaded is correct, but what reaches the tool is not. The auditor fetches the file over HTTP, so it sees what your server sends, not what is on your disk. Check these before you edit a single line.

• An HTML page in place of the file. Content management systems and single-page app servers often answer unknown addresses with the home page or a styled error page and a 200 status. A parser then finds <!DOCTYPE html> where the H1 should be. The fix is on the server, and the missing llms.txt guide covers it.
• The file wrapped in a page. Some site builders let you create a page called llms.txt, then wrap your text in their theme, with <p> tags and navigation. The Markdown is in there, but not as Markdown. Upload a real file or use a route that returns the raw text.
• A byte order mark. Some Windows editors save UTF-8 with three invisible bytes at the start. The first line then begins with those bytes rather than #, and a strict check for a line starting with a hash fails. Save as UTF-8 without BOM.
• The wrong encoding. A file saved as Windows-1252 turns pound signs, accented names and curly apostrophes into garbled characters when read as UTF-8. Save as UTF-8 and serve it with charset=utf-8.
• A download instead of a page. A Content-Disposition: attachment header or a type such as application/octet-stream does not stop a script from reading the bytes, but some tools treat it as binary. Serve text/plain or text/markdown.

curl -sS -D - https://www.example.co.uk/llms.txt -o llms-check.txt
head -c 3 llms-check.txt | xxd
file llms-check.txt

The response headers show the status and content type. If xxd prints efbb bf, the file starts with a byte order mark. If file reports HTML or an encoding other than UTF-8 or ASCII, fix that first.

A broken file and the same file corrected

This is the kind of file the auditor flags. It was typed into a website builder by the owner of a fictional garden centre, then partly pasted from a sitemap.

Brambleford Garden Centre
=========================

Welcome to Brambleford! We are the best garden centre in the county.

# Plants
• https://www.example.co.uk/plants/perennials
• https://www.example.co.uk/plants/shrubs

# Visiting us
### Opening times
[Opening hours](/visit/hours) - seasonal changes
1. [Cafe](https://www.example.co.uk/cafe/) (menu, allergens)

Counting the problems: a setext title, no blockquote, a marketing sentence as the only description, three H1 lines, typographic bullets with bare URLs, an H3 inside the link area, a relative link, a dash instead of a colon and a numbered item with its note in brackets. Here is the same content in the proposed structure:

# Brambleford Garden Centre

> Independent garden centre near Chesterfield selling plants, garden
> supplies and houseplants, with an on-site cafe. We do not sell online.

Prices include VAT. Plant stock changes weekly, so pages describe ranges
rather than individual plants.

## Plants
- [Perennials](https://www.example.co.uk/plants/perennials/): hardy border plants, when each range is in stock
- [Shrubs](https://www.example.co.uk/plants/shrubs/): evergreen and flowering shrubs, pot sizes, planting advice

## Visiting us
- [Opening hours](https://www.example.co.uk/visit/hours/): seasonal hours, bank holidays, late opening
- [Cafe](https://www.example.co.uk/cafe/): menu, allergen information, when food service stops

Nothing about the business changed. The file now has one title, a summary a parser can pick out, a free-text line of facts, and two sections whose every line is a complete entry.

The fix, step by step

A small checker you can run before uploading

This script needs nothing beyond Python 3. It reads a local file or a live URL and prints every structural problem it finds, using the same rules as the table above. It does not check that the links work; that is a separate job covered in the broken links guide.

import re, sys, urllib.request

src = sys.argv[1]
if src.startswith('http'):
    with urllib.request.urlopen(src) as r:
        ctype, raw = r.headers.get('Content-Type', ''), r.read()
    if 'html' in ctype:
        print('served as', ctype)
else:
    raw = open(src, 'rb').read()

if raw.startswith(b'\xef\xbb\xbf'):
    print('starts with a byte order mark')
text = raw.decode('utf-8', errors='replace')
if '�' in text:
    print('contains bytes that are not UTF-8')

lines = text.splitlines()
first = next((l for l in lines if l.strip()), '')
if not first.startswith('# '):
    print('first line is not an H1:', first[:60])
h1 = [l for l in lines if re.match(r'#\s', l)]
if len(h1) != 1:
    print('expected 1 H1, found', len(h1))

section = None
entry = re.compile(r'^- \[[^\]]+\]\(https?://[^)\s]+\)(: .+)?$')
for n, l in enumerate(lines, 1):
    if l.startswith('## '):
        section = l[3:]
    elif l.startswith('#'):
        if not l.startswith('# '):
            print(n, 'heading not allowed here:', l[:60])
    elif section and l.strip() and not entry.match(l.strip()):
        print(n, 'not a valid entry:', l[:70])
    elif section is None and re.search(r'\]\(|https?://', l):
        print(n, 'link outside any section')
print('done')

Save it as llmscheck.py and run python3 llmscheck.py llms.txt or python3 llmscheck.py https://www.example.co.uk/llms.txt. A clean file prints only done. The note is optional in the entry pattern, because the proposal makes it optional, so an entry without one passes this check. The auditor treats missing notes as a content weakness rather than a format error, so fill them in anyway.

Edge cases that confuse people

Linking to Markdown versions of pages

The proposal also suggests that sites offer a clean Markdown copy of useful pages at the same address with .md added. If you do that, the entries in llms.txt can point at those copies. It does not change the format of llms.txt itself, and if you have not built the copies, link to the normal HTML pages rather than to addresses that return errors.

Very long notes

A note can run to more than one sentence, but it must stay on the same line as the link. A line break in the middle of a note ends the entry, and the continuation line is then read as a stray line inside the section. If you need more than a sentence or two, the detail belongs on the linked page.

Section names

Apart from Optional, the proposal does not fix section names, so ## Docs, ## Services and ## Policies are all fine. Avoid naming a normal section Optional by accident, for example ## Optional extras on a shop, because some tools may treat everything under a heading starting with that word as skippable. Call it ## Add-on products instead.

Comments and front matter

Static site generators sometimes leave YAML front matter between --- lines at the top of a Markdown file, and developers sometimes leave HTML comments. Both push the H1 off the first line. Strip them from the published output.

Line endings and trailing spaces

Windows line endings and trailing spaces are harmless to the parsers we know of. You do not need to convert them, although many editors will tidy them automatically.

Keeping the format right after the fix

• Keep one master copy in plain text, in version control if you have it, and always upload from that. Never edit the live file through a rich text editor.
• Run the checker automatically as part of a build or deploy step, and stop the deploy if it prints anything other than done.
• Generate the link lines from data if your file is built by a script or plugin, so each entry comes out in the correct pattern every time. The WordPress guide and Shopify guide cover how the file is produced on those platforms.
• Re-run the auditor after theme changes, CMS upgrades and server moves, because those are when a correct file most often starts being served wrapped in HTML or with the wrong type.

Once the structure is sound, the next thing the auditor looks at is whether the notes are any use. That is a different problem, covered in the thin content guide.