Fixing a Stale llms.txt That No Longer Matches Your Site
An llms.txt file is usually written once, in an afternoon, and then forgotten while the site keeps changing. Months later it lists a service you dropped, quotes last year's prices, points at pages that now redirect and says nothing about the products that matter most today. That is a stale file.
A tool that reads it gets an outdated picture of your business. This guide shows how to measure the drift between the file and the live site, correct it properly, and set up a single source that rebuilds the file each time you publish.
What staleness looks like
A stale file is not the same as a broken one. The format can be perfect and every link can return a 200, yet the file still misleads, because the content moved on and the file did not. Drift shows up in five ways, and it is worth naming them because each needs a different check.
| Kind of drift | Example | How you spot it |
|---|---|---|
| Missing new pages | A new service page launched in spring is not listed | Pages in the sitemap that are important but absent from the file |
| Retired pages still listed | A discontinued product range, an old event page | Entries that 404, redirect or are now noindexed |
| Moved pages under old addresses | After a site rebuild, every URL gained a new folder | Entries that redirect, even if the redirect works |
| Notes that contradict the page | "Next-day delivery" when the page now says two to three days | Reading notes against the live pages |
| A summary that describes the old business | The blockquote still says "Chesterfield only" after you expanded to three counties | Reading the summary as if you were a new customer |
The first three are mechanical and a script finds them. The last two need a person, because only you know that the delivery promise changed. Broken and redirected links get their own treatment in the broken links guide; here they matter as symptoms that the file has fallen behind.
Why bother, given that llms.txt is only a proposal and no major AI search provider has confirmed reading it? Because a wrong file is worse than no file for the tools that do read it. A coding assistant or research tool that trusts your summary will repeat your old prices with confidence. If you are not prepared to keep the file current, deleting it is a reasonable choice.
Measuring drift against your sitemap
Your XML sitemap is usually generated automatically, so it tracks the site far better than a hand-written file does. Comparing the two gives you a quick list of candidates to add and entries to question. This script prints three lists: URLs in llms.txt that are not in the sitemap, the sitemap URLs changed since the file was last edited that are not in llms.txt, and the total count of each.
import re, sys, urllib.request, email.utils
from xml.etree import ElementTree as ET
site = sys.argv[1].rstrip('/')
def get(url):
with urllib.request.urlopen(url) as r:
return r.read(), r.headers
llms, h = get(site + '/llms.txt')
listed = set(re.findall(r'\]\((https?://[^)\s]+)\)', llms.decode('utf-8')))
edited = h.get('Last-Modified')
edited = email.utils.parsedate_to_datetime(edited).date().isoformat() if edited else '0000'
ns = {'s': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
xml, _ = get(site + '/sitemap.xml')
root = ET.fromstring(xml)
urls = {}
for sm in root.findall('s:sitemap/s:loc', ns): # sitemap index
for u in ET.fromstring(get(sm.text)[0]).findall('s:url', ns):
urls[u.findtext('s:loc', '', ns)] = u.findtext('s:lastmod', '', ns)
for u in root.findall('s:url', ns): # plain sitemap
urls[u.findtext('s:loc', '', ns)] = u.findtext('s:lastmod', '', ns)
print('llms.txt last modified:', edited)
print('\nListed but not in sitemap (moved, retired or noindexed?):')
for u in sorted(listed - urls.keys()): print(' ', u)
print('\nChanged since then and not listed (consider adding):')
for u, m in sorted(urls.items()):
if u not in listed and m[:10] > edited: print(' ', m[:10], u)
print('\n%d listed, %d in sitemap' % (len(listed), len(urls)))Run it with python3 llmsdrift.py https://www.example.co.uk. If your sitemap lives elsewhere,
such as /sitemap_index.xml on many WordPress sites, change the path. The second list will
be long on a busy site; you are not meant to add everything, only to scan it for pages a new customer
would need. The script depends on your server sending a Last-Modified header for llms.txt,
which static files normally have; if yours does not, it lists every unlisted page instead.
The fix, step by step
Step 1
List what the file says today
Download the live file with curl rather than trusting the copy on your computer, since the two often differ after months of small edits. Paste its entries into a spreadsheet with three columns: section, URL and note. Add a fourth column for the page title as it currently appears on the site. This one table becomes your working list, and later it becomes the source that generates the file, so take the time to fill it in properly.
Step 2
Compare it with the sitemap
Run the drift script in this guide against your domain. For every URL listed in llms.txt but absent from the sitemap, open it and find out whether it redirects, returns an error or carries a noindex tag, then replace it with the current address or remove it. From the list of recently changed pages that are not in the file, pick out any that a new customer would need and add them to the spreadsheet with a section.
Step 3
Check each note against its page
Open every remaining page next to its note and read both. Any fact in the note, such as a price, delivery time, service area, opening hours or product range, must match what the page says today. Where it does not, correct the note, or cut the fact out of it if it changes often enough that you would rather not repeat it. Then reread the blockquote summary at the top and bring it up to date as well.
Step 4
Move the page list into one source file
Save the cleaned spreadsheet as a CSV file named something like llms-pages.csv,
with columns for section, title, URL and note, and store the header text, meaning the H1, summary and
details, in a small text file beside it. Keep both with the site's code or content, not on someone's
desktop. From now on, nobody edits llms.txt by hand; they edit the source and regenerate, which is what
stops the file drifting again.
Step 5
Generate llms.txt from that source
Use
the generator script in this guide, or your CMS or plugin equivalent, to build llms.txt from the CSV
and header file. The generator writes the H1 and blockquote, groups the rows by section in a fixed order
with Optional last, and formats each row as a Markdown link with its note. Compare the output
with the file you downloaded in step one, upload it to the site root and run the llms.txt Auditor to
confirm it passes.
Step 6
Rebuild on every publish and check weekly
Add the generator to whatever runs when the site changes: the deploy script, the static site build, a CMS publish hook or, failing all of those, a nightly scheduled task. Then schedule the drift script to run weekly and email you its output. Regeneration keeps the format and links consistent, while the weekly report tells you when new pages have appeared that might deserve a place in the source list.
A generator that builds the file from a CSV
The source is deliberately simple so that anyone who can edit a spreadsheet can maintain it. Here
is a fragment of llms-pages.csv for a fictional dog grooming business:
section,title,url,note
Services,Full groom,https://www.example.co.uk/full-groom/,"bath, clip and nails, prices by breed size"
Services,Puppy introduction,https://www.example.co.uk/puppy/,"first visits for puppies under six months"
Booking,Book an appointment,https://www.example.co.uk/book/,"online diary, deposits, cancellation terms"
Optional,Before and after gallery,https://www.example.co.uk/gallery/,"photos of recent grooms"And llms-header.md holds the top of the file, exactly as it should appear:
# Pawsley Grooming Studio
> Dog grooming salon in Bakewell, Derbyshire. Appointments only,
> Tuesday to Saturday. We groom dogs, not cats.The generator reads both and writes the finished file:
import csv
from collections import OrderedDict
ORDER = ['Services', 'Booking', 'Policies', 'Optional']
rows = list(csv.DictReader(open('llms-pages.csv', encoding='utf-8')))
groups = OrderedDict((s, []) for s in ORDER)
for r in rows:
groups.setdefault(r['section'].strip(), []).append(r)
if 'Optional' in groups:
groups.move_to_end('Optional')
out = [open('llms-header.md', encoding='utf-8').read().strip(), '']
for name, items in groups.items():
if not items:
continue
out.append('## ' + name)
for r in items:
line = '- [%s](%s)' % (r['title'].strip(), r['url'].strip())
if r['note'].strip():
line += ': ' + r['note'].strip()
out.append(line)
out.append('')
open('public/llms.txt', 'w', encoding='utf-8', newline='\n').write('\n'.join(out))
print('wrote', sum(len(v) for v in groups.values()), 'entries')Change public/llms.txt to wherever your document root or static folder is. If you also
publish an llms-full.txt, build it in the same run from the same CSV so the two files can never disagree;
the llms-full.txt guide covers what goes
into that file.
Where to run the rebuild
The best trigger depends on how your site is published.
| How the site is published | Where the rebuild goes | Notes |
|---|---|---|
| Static site generator (Hugo, Eleventy, Astro, Jekyll) | As a step in the build command, before the output is uploaded | The source CSV lives in the repository and changes go through the same review as content |
| Framework app (Next.js, Nuxt) | A prebuild script, or a route that renders the file from the CSV or CMS at request time with caching | A generated route stays current without a redeploy, but test the content type it returns |
| Git-based deploy to a VPS or Plesk | A post-deploy hook that runs the generator on the server | Make sure the hook runs as a user that can write to the document root |
| WordPress or another database CMS | A plugin that regenerates on publish, or a scheduled task that queries published pages | See the WordPress guide for plugin options |
| Hosted shop or builder | The platform's own llms.txt feature or template, updated by hand from your source list | Shopify templates cannot loop over the catalogue; see the Shopify guide |
| Nothing automated at all | A monthly calendar reminder to run both scripts by hand | Better than nothing, but the first thing to slip when you are busy |
Whichever you choose, make the generated file the only version. If someone uploads a hand-edited llms.txt over the top, the next rebuild will quietly replace it, so tell everyone who edits the site where the source lives.
What automation cannot do for you
Generating from a source file fixes the mechanical drift, but three kinds of change still need a human decision.
Deciding what is important now
A generator lists what is in the source. It does not know that your new commercial cleaning service is now half your revenue and deserves its own section. Put a quarterly review in the diary: read the whole file as a stranger would and ask whether it still describes the business you run today.
Facts that change often
If notes contain prices, stock levels or dates, they will go out of date faster than any review cycle. Prefer notes that describe what the page covers, such as "prices by breed size", over notes that repeat the figures. Then the page carries the facts and the file only has to point to it.
Rebrands, mergers and domain moves
A change of business name or domain alters the H1, the summary and every URL at once. Treat it as a rewrite. After a domain move, keep serving an llms.txt on the old domain only if it redirects to the new one; a second, frozen copy on the old host is exactly the kind of stale file this guide is about.
Should the file carry a date?
The proposal does not define a date field, and the tools we know of do not look for one. The web server
already sends a Last-Modified header with static files, which is how the drift script above
works out when the file last changed.
Some site owners add a line in the free text area under the summary, such as This file was generated from the site content on 28 September 2026.
That is allowed by the format, because the details area may contain ordinary paragraphs. It can help
a human reader judge how current the file is. Only add it if the date is written by the generator, so
it is always true. A hand-typed date that nobody updates makes the file look staler than it is, and a
date that changes on every build without any content change tells the reader nothing.
1. How often should I update llms.txt?
Update it whenever the pages it lists change in a way that matters, which is why rebuilding automatically on publish is the best approach. On top of that, review the choice of pages every three months or so, and after any significant change to the business such as a new service, a price change or a move into new areas. A small site that rarely changes might need only two or three real edits a year.
2. Is an out-of-date llms.txt worse than having none?
It can be. A tool that reads the file trusts it as a summary written by the site owner, so an old price, a withdrawn service or a wrong service area may be repeated to someone as fact. Having no file just means the tool reads your pages directly. If you know you will not maintain the file, removing it is a sensible and honest choice, and the auditor will simply report it as missing rather than stale.
3. Can I generate llms.txt straight from my sitemap?
You can generate the list of URLs from a sitemap, but the result is rarely a good llms.txt. A sitemap holds every indexable page, with no notes and no sense of priority, so the file becomes long and unhelpful. A better pattern is to use the sitemap as a check, as the drift script in this guide does, while the list of pages and their notes lives in a smaller source file that a person curates.
4. Do AI tools cache old copies of llms.txt?
Some probably do, in the way any crawler or tool keeps copies of what it fetched, but providers have not published details of how they treat this particular file. You cannot force a refresh. What you control is serving the current version with correct headers, purging your own CDN cache after each rebuild, and keeping the address stable so that any tool that checks again finds the new content at the same place.
5. Should llms-full.txt be regenerated at the same time?
Yes. If you publish llms-full.txt, it holds the full text of your key pages, so it goes stale every time one of those pages is edited, not only when the list changes. Build both files in the same step from the same source list. That way they always agree about which pages matter, and the full text inside llms-full.txt always reflects the latest published version of each page rather than an old snapshot.
6. What if my CMS plugin generates llms.txt automatically?
Then most mechanical drift is handled for you, but check what the plugin actually does. Some regenerate on every publish; others only when you press a button or on a schedule. Some include every post, which produces the sitemap-style file this guide warns against. Look for settings that let you choose which post types or pages are included and edit the notes, and still read the output each quarter to confirm it describes the business correctly.
7. How does the auditor decide a file is stale?
The auditor checks the file against the proposed format, tests whether each linked URL responds and scores the quality of the content, so a stale file usually shows up as redirected or failing links and weaker content findings rather than one single flag. Treat any of those as a prompt to run the drift check in this guide, because they are often the visible edge of a wider mismatch between the file and the site.
The llms.txt Auditor fetches your /llms.txt and /llms-full.txt files, validates them against the proposed format, checks that each linked URL responds, confirms robots.txt allows the relevant AI crawlers and scores the quality of the content.
Open llms.txt Fixes →