llms.txt: What It Is and Why Your Website Needs One Now
Learn what llms.txt is, why it helps AI tools like ChatGPT and Perplexity understand your website, and how to create one in minutes. Step by step for complete beginners.
What you will learn in this guide
- What llms.txt is and where the standard comes from
- How llms.txt helps AI tools understand your website
- What to put in your llms.txt file
- How to check if your llms.txt is correctly formatted
- How to generate a llms.txt file for your site automatically
- How to verify AI tools can access your llms.txt
1 What is llms.txt?
llms.txt is a plain text file you place at the root of your website — just like robots.txt or sitemap.xml. It is designed to help AI language models (LLMs) like ChatGPT, Claude and Perplexity understand what your website is about and which pages are most important.
2 What goes in a llms.txt file
A basic llms.txt file contains your site name, a brief description, and links to your most important pages with descriptions of what each one covers.
# aiwebpageseo
> Free SEO tools for website owners — page speed, site audit, schema, content and local SEO.
## Core tools
- [Performance Tools](https://aiwebpageseo.com/performance-tools.html): Page speed, CLS, accessibility and HTTP error auditing
- [Audit Tools](https://aiwebpageseo.com/audit-tools.html): Site crawl, hreflang, redirects and meta analysis
- [Pricing](https://aiwebpageseo.com/pricing.html): Pay-as-you-go credits from 10p3 How to create and deploy your llms.txt
- 1Generate your llms.txt automaticallyGo to content-tools.html#llms and enter your site URL. The tool fetches your pages, scores your current llms.txt if one exists, and generates a corrected version.
- 2Upload the file to your server rootSave the generated content as llms.txt and upload it to the root of your site. It should be accessible at https://yoursite.com/llms.txt
- 3Verify it is accessibleVisit https://yoursite.com/llms.txt in your browser and confirm it returns plain text, not a 404 or HTML page.
- 4Update it when your site changesWhenever you add major new pages or change your site structure, re-run the LLMs.txt Auditor and update your file. Treat it like your sitemap — keep it current.
4 The honest status of llms.txt
Almost everything written about this file overstates it, so here is the position as it actually stands.
llms.txt is a community proposal. It is not a W3C or IETF standard. No assistant operator publishes a specification for it, and none has stated that it reads one. No relationship between publishing the file and being cited by anything has ever been demonstrated. The fact that a good number of websites have adopted it tells you about publisher enthusiasm, not about crawler behaviour — those are entirely different claims, and they get conflated constantly.
You will also see cohort statistics: sites with an llms.txt get cited more often. Treat those with suspicion. They do not control for the obvious confounder, which is that a team disciplined enough to publish and maintain an llms.txt is a team that maintains its content generally. The file did not cause the outcome. The same habit produced both.
5 llms.txt is not robots.txt, and neither is a permissions system
These two files get conflated constantly, and the confusion produces sites that believe they have controlled something they have not.
| robots.txt | llms.txt | |
|---|---|---|
| What it does | Governs access — tells named crawlers what they may fetch | Recommends content — says which pages are worth reading |
| Enforced? | By convention. Reputable operators honour it; it is a request, not an access control | Not at all. It prevents nothing and permits nothing |
| Can it block AI training? | Yes — by naming GPTBot, Google-Extended and the rest | No. A permissions statement inside it has no mechanism behind it |
Two corrections follow, and they are the ones that cost sites the most:
Blocking GPTBot does not remove you from ChatGPT. GPTBot is the training crawler. The agents that fetch pages to answer a live question are OAI-SearchBot and ChatGPT-User, and they follow their own directives. A site that blocks everything with "GPT" in the name usually intended to decline training and has accidentally removed itself from citations.
Google-Extended controls Gemini training only. It has no effect on Google Search, on your rankings, or on whether you appear in AI Overviews — those run on ordinary Googlebot access.
And whatever you write in either file, verify it in your access logs. A permissive robots.txt sitting behind a WAF rule that drops unfamiliar user-agents produces a flawless configuration and no crawler traffic at all.
6 Write it for the constraint, not for completeness
A generator will happily produce a file listing every page on your site. Resist that: an exhaustive llms.txt is a sitemap with prose attached, and it has thrown away the only thing that distinguished it, which was curation.
The question the file exists to answer is deceptively demanding: if somebody could read six pages of this site and no more, which six, and why each one? Most teams have never sat down and answered it, and the answer is rarely the navigation.
- Point at the pages that explain things. Documentation, guides, reference material. Explanatory prose survives being lifted out of a layout and read cold — which is exactly what a retrieval system does with it.
- Not at the pages that sell things. Marketing copy depends on the design around it, the image beside it, and a reader who already wants to believe you. Extracted as plain text it says almost nothing, which makes it the least useful thing you can nominate.
- One honest sentence each. What it contains, and who would want it. Not a keyword-loaded summary — nothing is scoring this.
7 Where the effort actually belongs
If the goal behind publishing llms.txt is to be usable by AI systems, then it is worth knowing where that goal is genuinely won and lost — because it is not in this file.
- 1Can the agents reach you?Confirmed in your access logs, not your robots.txt. A silent WAF block is the single most common cause of a site being invisible to assistants, and it produces no error anybody notices.
- 2Is the substance in the served HTML?Many retrieval fetchers parse rather than render. Content assembled client-side after load — prices, specifications, FAQ answers — does not exist for them. View source and search for your own text.
- 3Does your markup describe the page accurately?Structured data disambiguates: it tells a parser that this string is a price and that one is an author. It does not confer preference, and there is no verified evidence that denser schema wins citations.
- 4Has anybody written an answer worth quoting?The only item with unbounded upside. Assistants surface passages that stand up when lifted out of the page. Most marketing pages contain none — they contain persuasion, which collapses on extraction. Everything else on this list is a precondition. This is the work.
Publish the file. Keep it true. Then go and do the fourth thing, which is the one that would still be worth doing if llms.txt disappeared tomorrow.
8 Frequently asked questions
There is no evidence that any of them do. That is worth stating plainly, because almost everything written about this file assumes the opposite. llms.txt is a community proposal: no assistant operator publishes a specification for it, none has said it reads one, and no relationship between publishing the file and being cited has ever been demonstrated. Adoption by publishers is not the same as consumption by crawlers. Publish it because it is nearly free and the exercise is useful — not because it will do something for you.
Only in that both are plain text files at your site root, and the comparison misleads badly beyond that. robots.txt governs access: it tells named crawlers what they may fetch, and the reputable ones honour it. llms.txt makes no access claim at all — it is a curated recommendation about what is worth reading, and it prevents nothing, permits nothing and enforces nothing. Writing an AI-permissions statement inside llms.txt has no mechanism behind it and no established legal weight.
No. Training access is governed by robots.txt directives naming the relevant agents — GPTBot for OpenAI training, Google-Extended for Gemini training — and, if you need enforcement rather than a polite request, by a WAF rule. A sentence in llms.txt saying your content may not be used for training does not stop any training. Note too that training and retrieval are separate decisions: blocking GPTBot does not remove you from ChatGPT's cited answers, which are fetched by OAI-SearchBot and ChatGPT-User.
The pages you would want somebody to read if they had time for six of them and no more, each with one honest sentence saying what it contains and who it is for. That constraint is the whole discipline. Point at the pages that explain things rather than the pages that sell things — explanatory prose survives being lifted out of a layout and marketing copy does not. And resist being exhaustive: a file listing two hundred URLs is a sitemap with prose attached, which discards the only thing that made this format different, namely the curation.
Whenever the pages it points at change — which means it needs an owner, because this file has an unusual failure mode. When it rots, nothing breaks. The site keeps working, no monitor fires, no user complains, and a year later it is confidently recommending pages that were moved in March. A stale llms.txt is a machine-readable file on your own domain telling clients something false about your business. Regenerate it whenever documentation ships, and add its links to whatever already checks the site for broken URLs.
Four things, and llms.txt is not among them. First: can the retrieval agents fetch you? Check your access logs, not your robots.txt — a WAF rule matching unfamiliar user-agents produces a permissive config and zero traffic, silently. Second: is your substance in the served HTML? Many fetchers parse rather than render, so content assembled client-side after load does not exist for them. Third: is your structured data accurate, so a parser knows what your text means? Fourth, and by far the largest: has anybody written an answer worth quoting — a self-contained passage that still makes sense lifted out of the page?
Yes, on modest terms. It costs ten minutes, it does no harm, and if operators do converge on reading it you already have one. What you should not do is treat it as a lever, spend a week on it, or let it displace the four things above — which are the ones that actually decide whether a machine can read and cite your site. Publish it as cheap hygiene, keep it true, and put the real effort into the pages it points at.