Fixing Broken Links in llms.txt

What counts as a broken link here

A link can be "broken" for a reader in more ways than a 404. For llms.txt, the standard is stricter than for an ordinary web page, because the file is meant to be a hand-picked list of the best URLs on the site. Anything other than a direct, public, indexable page that answers with a 200 is worth fixing.

The first two are genuine dead ends. The rest are softer, but all of them either waste a reader's effort or point it at something you did not intend to recommend.

Why redirects are worth fixing

Most tools follow redirects, so a redirected link usually still reaches the page. It is still worth updating, for three practical reasons:

• It costs a request. A tool reading a file of forty links with a redirect on each makes eighty requests instead of forty.
• Not every client follows every redirect. Some fetchers refuse to follow a redirect to a different host, or stop after a small number of hops, to avoid being led somewhere unexpected.
• It signals that the file is out of date. A list of addresses that all redirect tells any reader the file was not written with the current site in view, which undermines the rest of it.

The most common redirect in llms.txt files is also the easiest to fix: the file was written with http://example.co.uk/page and the site now lives at https://www.example.co.uk/page/. Protocol, host name and trailing slash each cause a hop. Write every link exactly as the page's canonical tag states it.

Checking links by hand with curl

To see exactly what a tool sees for one link, ask curl for the status code, the number of redirects and the final address:

curl -s -o /dev/null -L --max-time 20 \
  -w "%{http_code} redirects=%{num_redirects} final=%{url_effective}\n" \
  https://www.example.co.uk/services/

A healthy result looks like 200 redirects=0 final=https://www.example.co.uk/services/. A result such as 200 redirects=2 means the page works but the link needs updating; 404 redirects=0 means the page is gone.

To run the same check over the whole file, extract the URLs from the Markdown links and loop over them:

curl -s https://www.example.co.uk/llms.txt \
 | grep -oE '\]\(https?://[^) ]+\)' \
 | sed -E 's/^\]\(//; s/\)$//' | sort -u \
 | while read -r url; do
     curl -s -o /dev/null -L --max-time 20 \
       -w "%{http_code} %{num_redirects} %{url_effective} <- $url\n" "$url"
   done

Any line that does not start with 200 0 needs attention. The loop uses a normal GET request rather than HEAD, because some servers answer HEAD requests with 405 or 404 even when the page is fine, which produces false alarms.

A fuller checker for noindex and canonical problems

Status codes miss the subtler problems: pages that answer 200 but carry a noindex directive, or whose canonical tag points at a different URL. This Python script, which needs the requests package, checks all three for every link in the file and prints only the entries that need work.

import re, sys, requests

FILE = sys.argv[1] if len(sys.argv) > 1 else "https://www.example.co.uk/llms.txt"
text = requests.get(FILE, timeout=20).text
urls = list(dict.fromkeys(re.findall(r"\]\((https?://[^)\s]+)\)", text)))
bad_md = re.findall(r"\]\((?!https?://)([^)]+)\)", text)

for u in bad_md:
    print("NOT ABSOLUTE    ", u)

for url in urls:
    try:
        r = requests.get(url, timeout=20, headers={"User-Agent": "llms-link-check"})
    except requests.RequestException as e:
        print("NO RESPONSE     ", url, type(e).__name__)
        continue
    problems = []
    if r.status_code != 200:
        problems.append("status %d" % r.status_code)
    if r.history:
        problems.append("redirects to " + r.url)
    html = r.text[:200000]
    robots = r.headers.get("X-Robots-Tag", "") + " " + " ".join(
        re.findall(r']+name=["\']robots["\'][^>]*>', html, re.I))
    if "noindex" in robots.lower():
        problems.append("noindex")
    m = re.search(r']+rel=["\']canonical["\'][^>]*href=["\']([^"\']+)', html, re.I)
    if m and m.group(1).rstrip("/") != r.url.rstrip("/"):
        problems.append("canonical is " + m.group(1))
    if problems:
        print("FIX             ", url, "|", "; ".join(problems))

Run it with python3 check_llms_links.py, or pass a local file path served through a test server. The canonical check is deliberately simple: it expects the rel attribute before href, so if your templates write them the other way round, adjust the pattern. Treat its output as a to-do list, not a verdict, and confirm each item in a browser.

What to change for each finding

A noindexed page is a judgement call. The file and the directive are sending opposite messages: one says "read this", the other says "do not put this in an index". Decide which is right. If the page is good enough to recommend, it is usually good enough to be indexed; if it is not, it does not belong in the file.

The fix, step by step

Edge cases that cause false alarms

• Bot protection. A CDN or security plugin may challenge scripted requests with a 403, a 503 or a JavaScript page while browsers pass. Check the same URL in a browser; if it loads, the link is fine but crawlers may have a problem, which the AI crawler access guide covers.
• Rate limiting. Checking a long file quickly can trigger 429 responses. Add a short pause between requests and re-test only the failures.
• Language or country redirects. A site that redirects based on the visitor's location or browser language will send your checker somewhere different from a crawler in another country. Link to the specific language version, and see the language guide for multilingual files.
• Fragments. A link such as /faq/#returns is fetched as /faq/; servers never see the part after the hash. The checker cannot tell whether the anchor still exists, so check those in a browser.
• Case and encoding. Paths are case-sensitive on most servers, so /About/ and /about/ may differ. Spaces and non-ASCII characters in URLs should be percent-encoded in the file.
• External links. Links to your own subdomains, a status page or a code repository are fine, but they break for reasons outside your site's control. Check them on the same schedule.

Stopping links breaking again

Links in llms.txt break for the same reason links anywhere break: pages change and nobody updates the places that point at them. The difference is that llms.txt is a single file you control, so it is easy to protect.

1. Generate the URLs, write the notes. If your CMS or build tool can produce the list of URLs from page IDs, links update themselves when slugs change, and you only maintain the notes.
2. Fail the build on a bad link. For static and framework sites, make the checker exit with an error when it finds a problem, so a broken entry never reaches production.
3. Treat llms.txt as an inbound link source during migrations. When you plan redirects for a redesign or a domain move, include every URL in the file in the redirect map and update the file on launch day.
4. Watch the auditor after big changes. Re-run it after a migration, a change of permalink structure or a new theme.

Broken links are one symptom of a file drifting away from the site. If whole sections now describe pages or services that no longer exist, the stale llms.txt guide covers rebuilding the file around the current site.