Fixing a Missing llms-full.txt: Build the Full-Text Companion File

What llms-full.txt is, and where it came from

The llms.txt proposal, published by Jeremy Howard in 2024, describes a small Markdown index at /llms.txt. It also suggests that sites offer clean Markdown versions of individual pages, and its reference tooling can expand an llms.txt file into a single context document containing the linked content. The specific name llms-full.txt is not defined in the original proposal text. It became a convention because documentation platforms and developer tools began publishing and reading a file with that name, holding the full text of a site's documentation in one place.

That history matters for two reasons. First, there is no formal specification for llms-full.txt, so the structure described below is common practice rather than a rule. Second, the tools that use it are mostly developer tools: coding assistants, agents and research tools that someone points at a documentation site. As far as we know, none of the major AI providers has confirmed that its search or answer products read llms-full.txt, or llms.txt, when choosing what to cite. Adding one is a convenience for tools that look for it, not a lever on AI visibility.

Does your site need one?

The honest answer for many small business sites is no. If your llms.txt lists eight pages of a few hundred words each, a tool can fetch those pages directly in seconds, and a second file adds maintenance without adding much. The file earns its place when a reader would otherwise have to follow dozens of links to understand the subject.

If you decide against it, that is a legitimate outcome. Leave the auditor finding as a known and accepted item, and put the effort into your main llms.txt file and the pages themselves instead.

What a well built file looks like

Because there is no formal specification, aim for a file a reader can navigate without guessing. The pattern most sites follow is the llms.txt header, then each page in turn with its title as a heading and its source URL stated, so a tool can cite the right page for each passage.

# Ledgerline Help Centre

> Help documentation for Ledgerline, bookkeeping software for UK sole
> traders. Covers the current web app only. Prices exclude VAT.

This file contains the full text of the pages listed in
https://help.example.com/llms.txt, generated on 2026-09-28.

## Connecting your bank

Source: https://help.example.com/bank-feeds/

Ledgerline imports transactions from your business bank account
through a secure bank feed. You choose which accounts to connect
and can disconnect them at any time...

### If a connection fails

...

## Making Tax Digital for Income Tax

Source: https://help.example.com/mtd-income-tax/

...

Several details in that example are deliberate:

• One H1 for the whole file, and each page starts at H2. The page's own headings are shifted down one level so the outline stays consistent.
• A source line under every page heading, giving the canonical URL. Without it, a tool that quotes a passage cannot link to where it came from.
• A generation date near the top, so a reader, or you, can see how current the file is.
• Only the main content of each page. Navigation, footers, cookie notices, related-post widgets and sign-up forms are removed.

Getting clean Markdown out of your pages

Extraction is where most generated files go wrong. A converter pointed at a whole HTML page will faithfully convert the menu, the footer and the newsletter box along with the article, so the same boilerplate appears once per page and can outweigh the real content. The fix is to convert only the main content element.

If your content already exists as Markdown

Static site generators such as Hugo, Jekyll, Eleventy, Astro and Docusaurus keep pages as Markdown or MDX source files. Build llms-full.txt from those files, not from the rendered HTML. Strip the front matter, resolve any include or shortcode syntax the generator would normally expand, and turn relative links into absolute URLs.

If your content lives in a CMS

Fetch the rendered page, select the main content element, usually <main> or <article>, and convert that fragment. Common converters include pandoc on the command line, markdownify or html2text in Python, and turndown in JavaScript. With pandoc:

curl -s https://help.example.com/bank-feeds/ -o page.html
pandoc -f html -t gfm --wrap=none page.html -o page.md

Pandoc converts whatever it is given, so cut the page down to its main element first, either with a small script or by using a CMS export or API that returns the body content alone.

Check these in the output

• Tables should come out as Markdown tables or readable lists, not as a jumble of cell text.
• Images should become their alt text, or be dropped. A line of image file names helps no one.
• Code blocks must keep their line breaks and indentation, fenced with three backticks.
• Accordions and tabs often hide text in the HTML with collapsed sections. Make sure the hidden text is included, since it is part of the page.
• Links inside the text should be absolute URLs, so they still work when read outside your site.

A small generator script

This Python example reads the links from your live llms.txt, fetches each page, keeps the <main> element, converts it with html2text and writes llms-full.txt. It needs the requests, beautifulsoup4 and html2text packages. Treat it as a starting point: adjust the selector to match your templates.

import re, datetime, requests
from bs4 import BeautifulSoup
import html2text

SITE = "https://help.example.com"
conv = html2text.HTML2Text()
conv.body_width = 0                       # do not wrap lines
index = requests.get(SITE + "/llms.txt", timeout=20).text

# Keep the H1, blockquote and details paragraph from llms.txt
header = index.split("\n## ")[0].strip()
urls = re.findall(r"\]\((https?://[^)\s]+)\)", index)

parts = [header, "",
         "Full text of the pages listed in %s/llms.txt, generated on %s."
         % (SITE, datetime.date.today().isoformat())]

for url in dict.fromkeys(urls):          # de-duplicate, keep order
    r = requests.get(url, timeout=20)
    if r.status_code != 200:
        print("skipped", r.status_code, url)
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    main = soup.find("main") or soup.find("article")
    if main is None:
        print("no main element", url)
        continue
    for tag in main.select("nav, aside, form, script, style, .share, .related"):
        tag.decompose()
    title = soup.title.get_text(strip=True) if soup.title else url
    body = conv.handle(str(main))
    body = re.sub(r"^(#+) ", r"#\1 ", body, flags=re.M)   # shift headings down
    parts += ["", "## " + title, "", "Source: " + url, "", body.strip()]

with open("llms-full.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(parts) + "\n")

The script prints any page it skips. Treat those messages as problems to fix in llms.txt itself, because a link that does not return a 200 response or has no main content is also a poor entry in the index. The broken links guide covers those cases.

The fix, step by step

Size, splitting and serving

A full-text file for a large documentation site can run to several megabytes. There is no agreed maximum, but bear in mind what readers do with it: a tool that loads the whole file into a model's context window is limited by that window, and a file much larger than it will be truncated or skipped. A few approaches keep size under control:

• Leave out Optional pages and anything that repeats content found elsewhere in the file.
• Split by product or section. A company with three products can publish one full-text file per product and link each from the relevant section of llms.txt.
• Offer per-page Markdown. The llms.txt proposal suggests making a Markdown version of each page available at the same URL with .md appended. Tools that want one page can then fetch just that page.

Headers to check

curl -sI --compressed https://help.example.com/llms-full.txt

Look for a 200 status, Content-Type: text/plain; charset=utf-8 or text/markdown; charset=utf-8, and a Content-Encoding such as gzip or br if compression is working. An HTML content type usually means a CMS or framework is serving an error page or a rewritten route instead of the file.

Search engines and duplicate text

The file repeats text that already exists on your pages. Search engines are generally good at recognising the HTML page as the main version, but if you would rather the text file did not appear in search results, send an X-Robots-Tag: noindex header for it. That header asks search engines not to index the file; it does not stop tools from fetching it, and nor should it.

Common problems the auditor turns up

• Boilerplate on repeat. The same menu and footer text appears once per page. Narrow the extraction to the main content element.
• A file that does not match the index. Pages listed in llms.txt are missing from the full file, or the full file contains pages the index does not mention. Generate from the index so the two cannot drift apart.
• Empty sections. A page built with client-side JavaScript returns almost no HTML to a script, so its section is blank. Generate from source content or from a rendered version instead.
• Stale text. The file was built once at launch and never again. Automate the rebuild.
• Relative links such as /pricing/ that mean nothing outside the site. Rewrite them as absolute URLs.
• Private content. Drafts, gated downloads or customer-only pages that should never have been public. Remove them, regenerate and check the source list.
• Wrong location. The file sits at /docs/llms-full.txt while llms.txt is at the root, and nothing links the two. Keep them on the same host, or link the full file explicitly.