llms.txt Fixes: Write a File Language Models Can Actually Use
An llms.txt file is a short Markdown document at the root of a website that tells large language models what the site is and which pages are worth reading. It is a proposal rather than an agreed standard, so it helps some tools and is ignored by others.
The llms.txt Auditor fetches your /llms.txt
and /llms-full.txt, checks them against the proposed format, tests every link, confirms
robots.txt lets AI crawlers in and scores the quality of the content. This page explains each finding
and links the guide that fixes it.
What llms.txt is, and what it is not
The llms.txt idea was published in September 2024 by Jeremy Howard of Answer.AI. The reasoning is simple: web pages are full of navigation, scripts, adverts and layout that get in the way when a language model tries to read them, and a model's context window is limited. A small, hand-written Markdown file could point the model straight at the pages that matter, with a sentence about each.
It is worth being clear about its status before spending time on it:
- It is not robots.txt. It grants and refuses nothing. Crawlers that are blocked in robots.txt stay blocked whatever llms.txt says, and pages left out of llms.txt can still be crawled.
- It is not a ranking signal we can point to. As far as we know, none of the major AI search providers has publicly said that it reads llms.txt when choosing which sources to cite. Some have been openly sceptical. Claims that it will lift your AI visibility are not backed by evidence.
- It is used by some tools. Documentation sites have adopted it widely, and some coding assistants and AI tools fetch it when pointed at a site, because a clean Markdown summary is easier to work with than rendered HTML.
So the honest case is this: an llms.txt file takes an hour or two to set up, costs nothing to host and may help tools that read it. It will not fix a site that AI crawlers cannot reach or read. If that is your problem, start with the Agent Readiness Guide.
The format the auditor checks against
The proposal defines a small, fixed structure in Markdown. The order matters, because tools that read the file expect to find the parts where the proposal puts them.
- One H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary: what the site is, who it is for and anything a reader must know to make sense of the rest.
- Optional paragraphs or lists with more detail, but no further headings in this part.
- One or more H2 sections, each containing a list of links. Each list item is a Markdown link, optionally followed by a colon and a note about the page.
- An optional H2 section called Optional, for links a tool can skip when it is short of space.
A small business site might look like this:
# Hartley & Rowe Joinery
> Bespoke kitchens, fitted wardrobes and staircases made in our
> Chesterfield workshop. We work across Derbyshire and South Yorkshire.
Prices on the site include VAT. We do not sell products online;
all work starts with a free survey.
## Services
- [Fitted kitchens](https://www.example.co.uk/kitchens/): design process, timber choices, typical lead times
- [Fitted wardrobes](https://www.example.co.uk/wardrobes/): alcove, sliding-door and walk-in options
- [Staircases](https://www.example.co.uk/staircases/): replacement and new-build, building regulations notes
## Customers
- [Book a survey](https://www.example.co.uk/survey/): how surveys work, areas covered, what to prepare
- [Frequently asked questions](https://www.example.co.uk/faq/): payment stages, guarantees, aftercare
## Optional
- [Project gallery](https://www.example.co.uk/gallery/): photos of finished work with short descriptions
Note what the file does not contain: no JSON, no HTML, no tracking parameters in the URLs and no marketing copy. Each note says what a reader will find on the page, not how good the business is.
/llms.txt
on the main host name. A file at /blog/llms.txt or only on the non-www version will not
be found by tools that look in the standard place.The fix guides
Each guide covers one auditor finding: what triggers it, how to put it right and how to confirm the fix.
Getting a valid file in place
Fixing a missing llms.txt gives a starter file you can adapt in minutes and explains where it must live. Fixing the file format covers the common structural mistakes: JSON or plain text instead of Markdown, several H1 headings, a missing summary blockquote, and links written as bare URLs without descriptions.
Making the content worth reading
Fixing thin content shows how to write useful notes for each link and how to choose which pages deserve a place. Fixing a missing llms-full.txt explains the companion file that holds the full text of your key pages in one document, and how to generate it rather than write it by hand. Fixing missing language declarations covers multilingual sites, where each link should make clear which language the page is in, or where separate files per language are clearer.
Keeping it accurate
Fixing broken links covers entries that return errors, redirect or point at pages marked noindex, and how to check them automatically. Fixing a stale llms.txt covers files that no longer match the site and how to rebuild the file whenever you publish. Fixing robots.txt blocking AI crawlers deals with the contradiction of publishing a guide for language models while refusing entry to the crawlers that would use it.
By platform
WordPress covers plugins that generate the file, uploading it by hand and multisite set-ups. Shopify covers the platform's limits on root files and the workarounds. Static sites covers generating the file at build time in frameworks such as Next.js, Astro and Hugo.
Choosing which pages to list
The most common weakness the auditor finds is not a broken file but an unhelpful one: either a handful of links with no notes, or a dump of every URL in the sitemap. Neither helps a model decide what to read. A good file is selective and explains itself.
| Type of site | Pages worth listing | Usually leave out |
|---|---|---|
| Local service business | Each service, areas covered, pricing or how quotes work, FAQs, contact and booking | Old news posts, tag pages, individual gallery images |
| Online shop | Main categories, delivery and returns, size guides, warranty, a few flagship products | Every product URL, filtered and sorted listing pages, basket and account pages |
| Software or SaaS | Product overview, documentation sections, API reference, pricing, changelog | Marketing landing pages that repeat the overview, sign-in pages |
| Publisher or blog | Topic hubs, cornerstone articles, editorial policy, about and author pages | Date archives, thin short posts, paginated lists |
Write each note for a reader who has never seen the site. "Delivery: UK mainland in 2 to 4 working days, Highlands and islands extra, returns within 30 days" is useful. "Learn more about our amazing delivery" is not. Keep facts in the note consistent with the page itself; if the page changes, the note must change too.
Which findings to fix first
- robots.txt conflicts. If the crawlers you want are blocked, the file cannot help them. Resolve this first, and make the decision about training and search crawlers deliberately.
- No file, or a file that cannot be fetched. A 404, a redirect to the home page, or
an HTML error page served at
/llms.txtall count as missing. - Format errors. A missing H1 or links that are not proper Markdown links can stop tools from reading the structure at all.
- Broken or redirected links. A guide full of dead ends is worse than a short one. Every link should return a 200 response at its final URL.
- Thin notes. Once the file works, add a clear sentence to each link and tidy the sections.
- llms-full.txt and language. Add these when the main file is right, if your site suits them.
- Staleness. Automate regeneration so the file does not drift away from the site again.
A sensible order of work
- Run the llms.txt Auditor on your domain to see whether the files exist and what is wrong with them.
- Check robots.txt for rules that block the AI crawlers you want, and fix them first.
- List the 10 to 40 pages that best explain your business, grouped into a few sections.
- Write the file in the proposed structure: H1, summary blockquote, optional details, H2 sections of links with notes, then an Optional section.
- Upload it to the site root, open it in a browser to confirm it loads as text and re-run the auditor.
- If your key pages are mostly text, generate an llms-full.txt from the same list rather than writing it by hand.
- Add a step to your publishing or build process that regenerates both files and checks every link.
The llms.txt Guide covers the proposal in more depth. Because llms.txt only helps once crawlers can read your pages, the Robots and Sitemap Guide and the AEO Guide are worth reading alongside it.
1. Does Google use llms.txt?
Google has not said that it uses llms.txt for Search or AI Overviews, and a Google representative has publicly compared it to the old keywords meta tag, which search engines stopped relying on long ago. Google Search works from its normal crawl of your pages. If Google visibility is your goal, llms.txt is not the place to start. Add it for the tools that do read it, and keep your expectations modest.
2. What is the difference between llms.txt and llms-full.txt?
llms.txt is an index: a short list of your key pages with a note about each. llms-full.txt holds the actual content of those pages, converted to plain Markdown and joined into one document, so a tool can load everything in one request instead of following links. The full file is most useful for documentation and reference sites. For a small business site with a handful of pages, the index alone is usually enough.
3. Where exactly should the file go?
It belongs at the root of your main domain, so it loads at an address like https://www.example.co.uk/llms.txt. It should return a 200 status, be served as text rather than HTML and not redirect somewhere else. If your site answers on both the www and non-www versions, make sure the version you treat as canonical serves the file. Subfolder locations are not where tools look by default, so a copy at /docs/llms.txt or /blog/llms.txt will usually go unnoticed unless the root file links to it.
4. Can llms.txt stop AI companies using my content?
No. llms.txt is a guide to your content, not a set of permissions. To ask crawlers not to collect your pages you need robots.txt rules for their named user agents, and to enforce that against crawlers that ignore robots.txt you need firewall or CDN rules. Leaving a page out of llms.txt does not hide it; it just means the file does not point anyone to it. Treat the file as a public signpost that anyone, human or machine, can read.
5. How long should an llms.txt file be?
There is no fixed limit, but the whole point is to be shorter and clearer than the site itself. Most small and medium sites are well served by somewhere between ten and forty links in a few sections, each with a one-line note. Very large sites can link to section-level pages or separate files rather than listing everything. If the file starts to look like your sitemap, it has lost its purpose, so trim it back to the pages you would show a new customer.
6. Should I write llms.txt by hand or generate it?
Write the structure and notes by hand at first, because choosing which pages matter and describing them clearly is a judgement a generator cannot make well. Then automate the maintenance: a build script or plugin can check links, add new pages to the right section and rebuild llms-full.txt when content changes. A file written once and never updated drifts out of date quickly and will start producing broken-link findings in the auditor within a few months.
7. Does llms.txt help with ChatGPT, Claude or Perplexity?
None of those providers has, as far as we know, publicly confirmed that its search or answer features read llms.txt when choosing sources. Their crawlers read your normal pages, so crawler access and clear page content matter far more. Where llms.txt can help is when someone uses an AI tool to explore your site directly, or with coding and research tools that look for the file. Its value may grow if adoption spreads.
The llms.txt Auditor fetches your /llms.txt and /llms-full.txt files, validates them against the proposed format, checks that each linked URL responds, confirms robots.txt allows the relevant AI crawlers and scores the quality of the content.
Run the llms.txt Auditor →