Adding an llms.txt File: When It Helps and How to Serve It

When the Agent Readiness Audit reports that a site has no llms.txt, it is flagging a missing extra, not a fault that stops AI crawlers reading your pages. The file is a proposal: a short Markdown summary at the root of a site that points language models at the pages worth reading.

Some tools use it, and no major AI search provider has confirmed that it relies on it. This guide helps you decide whether the file is worth your time, choose what belongs in it, publish it so that it is actually reachable, and check your logs to see whether anything requests it.

By John Carlsson · Updated 28 September 2026 · About 14 minutes to read

📋 Demo 📘 Tutorial 📖 Guide 📚 Book 🔧 Fix

In this guide

Where llms.txt fits in agent readiness
Is it worth adding to your site?
Choosing what goes in the file
A worked example
Serving the file so it can actually be fetched
The fix, step by step
Finding out whether anything reads it
Mistakes that make the file worse than none
Questions and answers

Where llms.txt fits in agent readiness

Every other finding in the Agent Readiness Audit is about whether a crawler can reach, render and understand your real pages. llms.txt is different. It adds a separate, optional document that sits beside those pages and describes them. If the pages themselves are blocked, empty without JavaScript or confusing to extract, a tidy llms.txt does not change that; it just points to pages the crawler still cannot use.

It helps to be precise about what is known:

It is a published proposal. The format is described at llmstxt.org. It is not an internet standard, and no standards body maintains it.
Adoption is uneven. Documentation platforms and developer tools have taken it up most readily. Some coding assistants and agent frameworks will read an llms.txt if they are pointed at a site.
Use by AI search is unconfirmed. As far as we know, none of the major AI search or assistant providers has publicly stated that it reads llms.txt to choose which sources to cite. Treat claims that it raises AI visibility with caution.
It grants and refuses nothing. It is not an access control file. Only robots.txt and your server decide what a crawler may fetch.

So the reasonable position is modest. The file is cheap, low risk and occasionally useful. It belongs at the end of your agent readiness work, not the start.

💡 If you want the detail of every format rule and the companion llms-full.txt file, the llms.txt fixes cover each one. This guide concentrates on the agent readiness side: deciding, publishing and checking.

Is it worth adding to your site?

The value depends on what kind of site you run and who is likely to point a tool at it.

Type of site	Likely value	What to put in it
Software documentation or an API	Highest. Developers do paste documentation sites into coding assistants.	Getting started, authentication, the main reference sections, changelog, limits.
Service business or trades	Low but cheap. A clear summary of what you do and where can still help a tool that reads it.	Services, areas covered, pricing approach, how to book, contact details.
Online shop	Modest. Listing products individually is pointless for large catalogues.	Category pages, delivery and returns, size guides, warranty terms.
Publisher or blog	Modest. Content changes too often for a hand-kept list.	Evergreen explainers, topic hubs, editorial and corrections policy.
Site behind a login	Very little. Tools cannot follow links into pages they cannot open.	Public help pages only, if any.

If your site falls in the first row, spend real time on it. For most small business sites, an hour to produce a good first version is the right investment, and no more.

Choosing what goes in the file

The file is a reading list, not a sitemap. Your XML sitemap already lists every URL for crawlers; llms.txt should do something a sitemap cannot, which is to tell a reader which few pages answer the questions people actually ask, and what each one contains.

Good candidates

Pages that state facts customers ask about: prices or how pricing works, delivery areas, opening hours, guarantees, lead times.
Pages that pass the rest of the audit, with full text in the HTML the server sends. A page that is empty without JavaScript is no more useful to a tool because it is listed.
Your about page and contact page, so a reader can tell who stands behind the site.
Policies with legal or practical weight, such as returns, terms and privacy.

Leave out

Pages that are noindexed, redirected, thin or out of date.
Tag archives, search results, paginated listings and filter URLs.
URLs with tracking parameters, and any address that is not the canonical version of the page.
Marketing claims. Notes should describe what is on the page, not how good the business is.

Twenty to forty links is plenty for most sites. A documentation site may need more, grouped into clear sections, with less important material placed under an ## Optional heading so a tool short of space can skip it.

A worked example

Here is a complete file for a fictional software company whose help centre is the main thing people want to read. It follows the proposal's order: one H1, a blockquote summary, a short paragraph, then H2 sections of links with a note after each.

# Tidewater Bookings

> Online appointment booking for UK salons, clinics and studios. Customers
> book and pay deposits through a widget on the business's own website.

Prices on this site are in pounds and exclude VAT. The help centre covers the
current version only; older versions are not documented.

## Getting started
- [Set up your account](https://help.example.com/start/): first login, business details, opening hours
- [Add the booking widget](https://help.example.com/widget/): embed code, WordPress and Squarespace instructions
- [Take deposits](https://help.example.com/deposits/): connecting a payment provider, refund rules

## Running the diary
- [Staff and rotas](https://help.example.com/staff/): adding staff, working patterns, holidays
- [Reminders](https://help.example.com/reminders/): email and text reminders, timing, wording
- [Cancellations and no-shows](https://help.example.com/no-shows/): policies, fees, automatic handling

## Account
- [Plans and pricing](https://www.example.com/pricing/): monthly plans, what each includes
- [Data and privacy](https://www.example.com/privacy/): where data is stored, retention, exports

## Optional
- [Release notes](https://help.example.com/releases/): changes by month

Each note names what the reader will find, in a few words. The summary states facts that would change how the rest is read: the audience, the currency and the VAT position. Nothing in the file is a claim that could not be checked against the linked page.

Serving the file so it can actually be fetched

A surprising number of files that exist are still unreachable. The audit can only credit a file that answers at the expected address with the expected response, so check these points.

Exact location. https://www.example.co.uk/llms.txt on the host name your canonical URLs use. If both www and non-www exist, the other one should redirect to it in a single hop.
Status 200. Not a redirect to the home page, not a soft 404 that returns your HTML error page with a 200 status.
Plain text content type. text/plain; charset=utf-8 is the safest choice. text/markdown is also reasonable. text/html suggests the file is being wrapped in a page template.
No robots.txt block. Rules such as Disallow: /*.txt$ are sometimes added to hide text files and catch llms.txt too.
No firewall challenge. If your CDN challenges AI crawlers, it challenges them on this file as well.

curl -sI https://www.example.co.uk/llms.txt | grep -iE "^HTTP|^content-type|^location"
curl -s -A "ClaudeBot" https://www.example.co.uk/llms.txt | head -5

The first command should show a 200 and a text content type with no location line. The second should print your H1 and summary, not an HTML page or a challenge.

Platform notes

Custom or static sites. Put the file in the web root or your static folder. If nginx serves it as application/octet-stream, add a rule such as location = /llms.txt { default_type text/plain; charset utf-8; }.
WordPress. Upload the file to the directory that holds wp-config.php, or use an SEO plugin that generates one. Check that a caching or security plugin is not rewriting the address.
Shopify. Files cannot be uploaded to the store root, so a workaround is needed; the Shopify llms.txt guide covers the options.

The fix, step by step

Step 1

Fix the access and content findings first

Open your Agent Readiness Audit report and deal with crawler blocks, empty server HTML, facts locked in images and canonical conflicts before you touch llms.txt. Those findings decide whether any AI system can use your pages at all. A reading list that points at blocked or empty pages achieves nothing, so treat this file as the last item on the list rather than a quick win to tick off first.

Step 2

Pick the pages a reader should start with

List the questions customers or users ask most often, then find the one page that answers each. Add the about, contact and key policy pages. Check every candidate: it should return 200 at its canonical address, carry no noindex, and show its main text with JavaScript switched off. Drop anything that fails. For most small sites you will end up with somewhere between twenty and forty links, grouped by what a reader is trying to do.

Step 3

Draft the file in plain Markdown

Start with a single H1 holding the site or business name, then a blockquote of one to three sentences that states what you do, for whom and anything that changes how the rest should be read, such as currency or service area. Add H2 sections of Markdown links, each followed by a colon and a short factual note. Put secondary material under ## Optional. Use absolute canonical URLs with no tracking parameters.

Step 4

Publish it at the root as plain text

Save the file as UTF-8 and upload it so that it answers at /llms.txt on your main host name. On WordPress that means the folder holding wp-config.php or a plugin that generates it; on static and custom sites, the web root or static folder. Configure the server to send text/plain; charset=utf-8 and make sure no page template, redirect or caching rule wraps the file in HTML.

Step 5

Check it is reachable by crawlers

Run curl -sI against the address and confirm a 200 status, a text content type and no redirect. Fetch it again with a crawler token such as ClaudeBot as the user agent to make sure the firewall is not challenging it. Check robots.txt for any rule that would match /llms.txt, then open every link in the file and confirm none redirect or fail. Finally, re-run the Agent Readiness Audit.

Step 6

Set a trigger for keeping it current

Decide what event will prompt an update: a new service, a price change, a page removed or a site restructure. Add the file to whatever checklist you use when publishing, or generate it from your CMS so it cannot drift. Every few months, search your access logs for requests to /llms.txt and note which user agents fetch it, so you know whether the effort is being read by anything.

Finding out whether anything reads it

Because use of the file is unconfirmed for most AI services, your own logs are the only evidence that matters for your site. After publishing, look for requests to the file and group them by user agent:

grep "GET /llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

What you see will vary. You may find SEO tools and audit services, scripts run by developers, general crawlers that request every text file they see, and occasionally an AI-related agent. Requests for the file are not proof that its contents shaped any answer; they only show it was fetched. If after several months nothing but audit tools has asked for it, that is useful information too, and a reason not to spend more time on it than a periodic check.

⚠️ Do not add tracking parameters to the links inside llms.txt to measure clicks. They create duplicate URLs, which is one of the problems the audit's canonical checks look for.

Mistakes that make the file worse than none

Using it like robots.txt. Lines such as Disallow: /private/ or Allow: GPTBot have no meaning in llms.txt. If you want to control access, do it in robots.txt and the firewall.
Contradicting the site. Prices, service areas or opening hours in the summary that differ from the pages give any reader two versions of the facts. Keep changeable figures on the pages and describe them in general terms in the file.
Dumping the sitemap. Thousands of links without notes are no easier to use than the sitemap itself.
Welcoming crawlers you block. A file written for language models while robots.txt disallows the crawlers that would fetch it sends mixed signals. Make the two agree.
Writing it once and forgetting it. Links to deleted pages are the most common fault in older files.

For how the file relates to the rest of your AI crawler set-up, read fixing AI crawler access, and for the first-file walkthrough in the llms.txt tool's terms, see fixing a missing llms.txt.

Questions and answers

1. Do ChatGPT, Claude or Perplexity use llms.txt?

None of the major providers has publicly confirmed that its search or answer systems read llms.txt when deciding which sources to use or cite, as far as we know. Individual tools built on these models, such as some coding assistants and agent frameworks, may fetch the file when a user points them at a site. Check your own access logs to see what actually requests it, rather than relying on claims made in marketing material.

2. Will adding llms.txt improve my rankings or AI citations?

There is no evidence that it will. It is not a Google ranking signal, and no major AI search provider has said it uses the file to pick sources. What does affect whether AI systems can use your content is crawler access, server-rendered text, clear structure and consistent facts, which are the other findings in the audit. Add llms.txt because it is cheap and may help some tools, not because you expect a measurable lift.

3. Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which paths they may fetch, and the major crawlers follow it. llms.txt tells a reader which pages are worth reading and what each contains. It cannot allow or block anything. If a crawler is disallowed in robots.txt, listing pages in llms.txt does not let it in, and leaving a page out of llms.txt does not keep it out. Keep access rules in robots.txt and your firewall.

4. What content type should llms.txt be served with?

Serve it as text/plain with a UTF-8 character set, which every client handles. text/markdown is also a sensible choice because the file is Markdown. Avoid text/html, which usually means a CMS is wrapping the file in a page template, and application/octet-stream, which some servers use for unknown extensions and which makes browsers download the file instead of showing it. Check the header with curl after publishing, and again after any server or CDN change, because a new configuration can quietly alter it.

5. How many links should an llms.txt file contain?

Enough to cover the questions people actually ask, and no more. For a small business that is often twenty to forty links in a handful of sections. A documentation site may need more, but should still group them clearly and move secondary material under an Optional heading. If the file is starting to look like your XML sitemap, you are listing too much, because a reader gains nothing from a list with no selection.

6. Do I need llms-full.txt as well?

It is optional. llms-full.txt is a companion convention in which the full text of key pages is combined into one Markdown document, and it is most common on documentation sites where tools benefit from reading everything at once. For a service business or shop it is rarely worth the upkeep, because it duplicates page content that then has to be kept in step. Start with a good llms.txt and add the full version only if you have a clear reason.

7. Can llms.txt cause any harm?

Very little, if it is accurate. The risks are indirect: a file that lists outdated prices or deleted pages gives any reader wrong information, links with tracking parameters create duplicate URLs, and a file that welcomes language models while robots.txt blocks their crawlers sends mixed signals. It also cannot keep anything private, so never list pages you would not want read. Keep it short, factual and in step with the site.

Check your llms.txt alongside everything else

The Agent Readiness Audit checks robots.txt and llms.txt, requests your page with AI crawler user agents and reviews how readable the content is, so you can see where the file fits among the other findings.

Open Agent Readiness Fixes →