llms.txt: AI Visibility File Setup in Under 10 Minutes

What a good llms.txt looks like

A well-structured llms.txt file has four sections: your site name as an H1, a brief description prefixed with >, an optional extended description, and a list of your most important pages grouped by section.

How to create and deploy your llms.txt

Option 1 — Generate automatically

Run the LLMs.txt Auditor — it crawls your site, identifies your key pages, and generates a ready-to-deploy llms.txt file tailored to your site structure. Download and upload to your server root.

Option 2 — Write manually

Create a plain text file named llms.txt. Follow the format above. List your 10-20 most important pages with clear descriptions. Upload to your server root so it is accessible at yoursite.com/llms.txt.

Verify deployment

Visit yoursite.com/llms.txt in a browser. It should display plain text content — not a 404 page or formatted HTML. Check your server returns Content-Type: text/plain for the file.

The spec, precisely — because most llms.txt files are not valid

llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI, and the specification lives at llmstxt.org. It is short, and it is stricter than most implementations realise. A file that "looks about right" frequently is not.

The ## Optional section is not a suggestion

This is the part almost everyone gets wrong. ## Optional is a reserved section name with a defined meaning: it marks links that a consumer may skip when it needs a shorter context. It is not "a section for optional extras" and it is not a synonym for "further reading".

It follows that ## Optional must be the last section in the file. A file with three sections after it has told a machine to discard content it then presents as important, which is incoherent. If a link matters, it does not belong under ## Optional.

Descriptions must be true

The notes after each link are the whole point of the format — they are what distinguishes it from a sitemap. A description that overstates what a page contains does not gain you anything: it produces a citation for a claim the page does not support, which is worse than no citation. Describe what is actually on the page.

What llms.txt does not do — and the claims to disregard

This file has attracted a large amount of marketing nonsense, and believing any of it leads to real mistakes.

• It is not access control. llms.txt grants nothing and blocks nothing. It cannot prevent your content being used for training, and it cannot stop any crawler fetching anything. Access is governed by robots.txt and by what your server actually serves. Anyone selling llms.txt as an opt-out is selling a fiction.
• It is not a ranking factor. Not in Google, not anywhere. No search engine has stated that it uses the file as an input to ranking.
• Support is not confirmed by the major AI vendors. Be clear-eyed about this: no large search or assistant provider has publicly committed to reading llms.txt. Adoption claims circulating in the SEO industry are, for the most part, inference rather than confirmation.
• It will not rescue a site that AI crawlers cannot read. If your content is rendered by JavaScript, gated behind consent, or blocked to the relevant user agents, a text file pointing at it changes nothing at all.

So why publish one?

Three honest reasons, none of which depend on a vendor announcement:

1. It costs almost nothing. A curated file of twenty links, written once and maintained occasionally, is a trivial investment against a possible future payoff.
2. The exercise itself is valuable. Being forced to name the twenty pages that actually matter, and to write one honest sentence about each, is a content audit in disguise. Most sites cannot do it, and discovering that is worth the afternoon on its own.
3. Clean, structured, plainly-stated content is what retrieval systems consume — whatever file they arrive through. The file is a small bet; the discipline behind it is not.

What actually determines whether an AI cites you

If the goal is being quoted in an AI answer, the file at your root is a long way down the list of things that matter. These are the things that decide it, roughly in order.

1. Can the crawler fetch you at all?

Every AI system has its own user agents, and they do different jobs. Blocking them is a legitimate choice — but it must be a choice, not an accident inherited from a robots.txt somebody copied in 2019.

2. Is the answer in the HTML?

Many AI retrieval systems fetch and parse rather than executing a full browser render. Content that only exists after JavaScript has run may simply not be seen. Server-render anything you want quoted. This is the single largest technical determinant of AI visibility and it has nothing to do with llms.txt.

3. Is the answer stated plainly, near the top, in one place?

A retrieval system is looking for an extractable passage that answers a question. A page that states the answer in a direct sentence under a question-shaped heading is trivially quotable. A page that builds to its point over eight paragraphs of throat-clearing is not — the answer exists, but it cannot be lifted out.

4. Are you corroborated elsewhere?

Answer engines favour claims they can see supported in more than one place. Being cited, referenced and linked by other sites is as central to AI visibility as it has always been to search. There is no file you can publish that substitutes for it.

Deployment: the failures that make the file worthless

The page above covers creating and uploading the file. These are the things that then quietly break it — every one of them common, every one invisible unless checked.

• The server returns HTML. A single-page-application host or a framework catch-all route will happily serve your app shell at /llms.txt with a 200 status. It looks fine in a browser tab. It is not a text file. Check the response, not the render.
• A link in the file 404s. A file pointing at pages that no longer exist is actively misleading — it is a curated list of your most important content, and some of it is gone. Re-validate every URL whenever the file is regenerated.
• A link in the file is blocked in robots.txt. This is a direct self-contradiction: you have told a machine that a page is among your most important, and separately forbidden it from fetching that page. Every URL in llms.txt must be crawlable.
• The file contradicts the site. Sections that no longer exist, a product you discontinued, a description written for a page that has since been rewritten.
• It is generated once and never touched again. This is the default outcome. Put its review on the same schedule as your sitemap, or accept that it will be wrong within a year.

A note on llms-full.txt

A separate convention has grown up around llms-full.txt: rather than links to your content, it contains the content itself, concatenated as markdown. It is not part of the core specification and is best understood as a related practice. It suits documentation and reference material, where a machine benefits from having the whole corpus in one fetch. It suits a marketing site considerably less — and a very large file has its own costs, both to generate and to consume.

Frequently asked questions

What is llms.txt?

llms.txt is a plain text file placed at the root of your website (yoursite.com/llms.txt) that provides AI language models with a structured guide to your site. It follows a Markdown-like format with your site name, a brief description, and links to your most important pages with descriptions of what each covers. It is the AI equivalent of a sitemap.xml.

Which AI tools read llms.txt?

Be clear-eyed about this: no major search or assistant provider has publicly committed to reading llms.txt, and claims of widespread adoption circulating in the SEO industry are inference rather than confirmation. There is no published evidence that the file improves citation rates. The honest case for publishing one does not rest on vendor support — it is that the file costs almost nothing, that curating twenty links and one honest sentence each is a content audit in disguise, and that the clean, plainly-stated content the exercise produces is what retrieval systems consume however they arrive at it.

How is llms.txt different from robots.txt?

robots.txt tells crawlers which pages NOT to visit. llms.txt tells AI systems which pages ARE most important and what they cover. They serve complementary purposes. robots.txt is for access control. llms.txt is for content guidance. Both should be present on a well-optimised site.

Does llms.txt stop my content being used for AI training?

No. llms.txt grants nothing and blocks nothing — it is content guidance, not access control. It cannot prevent your pages being crawled or used, and anyone presenting it as an opt-out is mistaken. Access is governed by robots.txt and by what your server chooses to serve.

What does the ## Optional section mean?

It is a reserved section name with a defined meaning in the specification: it marks links a consumer may skip when it needs a shorter context. It does not mean "optional extras" or "further reading". Because its contents are the first thing to be discarded, it must be the last section in the file, and anything you actually want read should not be in it.

Why is my llms.txt file not working?

The most common causes are that the server is returning HTML rather than plain text — a framework catch-all route will serve your app shell at that path with a 200 status and it looks fine in a browser — or that the URLs listed in the file 404, or are themselves blocked in robots.txt, which is a direct self-contradiction. Check the response, not the rendered page, and re-validate every URL each time the file is regenerated.

What matters more than llms.txt for getting cited by AI?

Four things, all ahead of it. Whether the relevant crawler is permitted to fetch you at all — noting that the agent retrieving a page to answer a live question is usually distinct from the one gathering training data, so blocking the wrong one removes you from answers rather than from the training set. Whether the content exists in the served HTML rather than only after JavaScript runs. Whether the answer is stated plainly and near the top, in a passage that can be extracted. And whether other sites corroborate you.

Related tools