How to Set Up llms.txt in Static Sites (Astro, Hugo, Eleventy)

Static-site generators (Astro, Hugo, Eleventy, Gatsby) make llms.txt clean: generate the file at build time from your content data, serve from /llms.txt root. This guide covers SSG-specific llms.txt. Pair with llms.txt guide and the WordPress version.

Step-by-step: How to set up llms.txt in static sites

1. Pick content to include. Curated list of your most valuable content: top blog posts, comprehensive guides, About/FAQ. 20-100 entries. Not full sitemap (different file).
2. Astro: create src/pages/llms.txt.ts. Astro supports custom endpoints. Create src/pages/llms.txt.ts that exports a GET function returning Response with text/plain content. Build collects all posts via getCollection() → builds markdown output.
3. Hugo: create layouts/_default/llms.txt and content/llms.md. Hugo: content/llms.md with content. layouts/_default/llms.txt template loops over .Site.RegularPages → outputs markdown lines. Configure mediatypes in config.toml to add text/plain. Output goes to public/llms.txt at build.
4. Eleventy: create llms.njk template. src/llms.njk: --- permalink: /llms.txt --- followed by Nunjucks template iterating collections.all → outputting markdown lines. Set permalink to ensure file generates at /llms.txt root.
5. Gatsby: create gatsby-node.js custom file. gatsby-node.js → onPostBuild hook → use fs.writeFileSync to write public/llms.txt with content built from GraphQL queries.
6. Validate file at /llms.txt. After build: visit yoursite.com/llms.txt. Should be text/plain (not text/html). Should contain markdown content. Test from curl: curl -I yoursite.com/llms.txt → Content-Type should be text/plain.
7. Monitor. Periodically (monthly) review llms.txt output is current with site content. Static generation means file is rebuilt on every deploy — stale only if you forget to deploy.

Before you build this: what llms.txt actually is

The tooling below is sound and the build-time approach is the right one. It is worth being honest about what you are building, because the guides that surround this topic are not.

llms.txt is a community proposal. It is not a W3C or IETF standard. No assistant operator publishes a specification for it, and none has stated that it reads one. No relationship between publishing the file and being cited by anything has ever been demonstrated.

You will see cohort statistics claiming sites with an llms.txt get cited more. Treat them carefully: they do not control for the obvious confounder, which is that a team disciplined enough to publish and maintain an llms.txt is a team that maintains its content generally. The file did not cause the outcome; the same habit produced both.

Build-time generation solves the only real problem

There is one failure mode that matters for this file, and it is not formatting. It is rot.

A hand-written llms.txt is accurate on the day it ships. Six months later it points at three pages that moved, one that was deleted, and describes a product that has changed. Nothing breaks. No monitor fires. No user complains. The file simply becomes a machine-readable document on your own domain telling clients something false about your business — and because it fails silently, nobody ever looks.

Generating it from your content collection at build time removes that failure entirely, and that is the actual argument for doing it this way:

• A deleted page cannot be listed, because the collection no longer contains it.
• A moved page updates its own URL, because the URL comes from the same source the site does.
• A changed description changes here too, because there is one source of truth.

This is the same principle that makes structured data reliable in a static site: derive it from your data, never maintain it by hand. Anything typed twice will diverge, and the divergence will be silent.

Curate ruthlessly, or you have just built a second sitemap

The temptation with build-time generation is to iterate every page in the collection, because it is one line of template code. Resist it. A file listing four hundred URLs is a sitemap with prose attached, and it has discarded the only thing that made this format different, which was curation.

The question the file exists to answer is demanding: if somebody could read six pages of this site and no more, which six, and why each one? Most teams have never sat down and answered that, and the answer is rarely the navigation.

Point at the pages that EXPLAIN things:
  - Documentation, guides, reference material
  - Prose that survives being lifted out of a layout
    and read cold — which is what retrieval does

NOT the pages that SELL things:
  - Marketing copy depends on the design around it,
    the image beside it, and a reader who already
    wants to believe you
  - Extracted as plain text it says almost nothing

In practice that means a filter, not a loop. Astro: getCollection('blog').filter(p => p.data.featured). Hugo: a page parameter, or a taxonomy term. Eleventy: a tag. The filter is the file’s whole value, and the loop without it is a waste of a build step.

llms.txt is not robots.txt, and neither is a permissions system

These two files sit at the same place and are constantly conflated, and the confusion produces sites that believe they have controlled something they have not.

robots.txt
  - Governs ACCESS: what named crawlers may fetch
  - Honoured by convention, by reputable operators
  - Can decline AI training: GPTBot, Google-Extended
  - It is a request, not an access control

llms.txt
  - RECOMMENDS content: which pages are worth reading
  - Enforces nothing. Permits nothing. Prevents nothing.
  - A permissions statement inside it has no mechanism
    behind it and no established legal weight

Two corrections follow, and they are the ones that cost sites the most:

Blocking GPTBot does not remove you from ChatGPT. GPTBot is the training crawler. The agents that fetch pages to answer a live question are OAI-SearchBot and ChatGPT-User, and they follow their own directives. A site that blocks everything with “GPT” in the name usually meant to decline training and has accidentally removed itself from citations.

Google-Extended controls Gemini training only. It has no effect on Google Search, on your rankings, or on whether you appear in AI Overviews — those run on ordinary Googlebot access.

And if you need genuine enforcement rather than a polite request, the answer is a WAF rule or authentication. No text file at your root has ever stopped anybody who did not want to be stopped.

llms-full.txt: know what you are shipping

The FAQ below notes you can generate an llms-full.txt containing full article bodies rather than links. The pattern is straightforward at build time; the decision is not, and it deserves a sentence of thought before the template gets written.

You are publishing your entire content library as a single plain-text file, pre-extracted and pre-cleaned, at a predictable URL. That is a convenience for anything that wants to consume it — and “anything” includes clients you have not thought about, since the file is a public URL like any other and honours no permissions of its own.

Reasonable if:
  - Your content is already fully public
  - You would be content for any of it to be quoted
  - The file stays current (build-time generation)

Think twice if:
  - Content sits behind any kind of gate
  - You are relying on people reaching the page
    to see the CTA, the context, or the offer
  - The file will be generated once and forgotten

And watch the size. A file of several hundred kilobytes is fine to serve and unpleasant to regenerate on every build in a large site — and a stale llms-full.txt is worse than none, because it is a complete, confident, machine-readable copy of a version of your site that no longer exists.

Where the effort actually belongs

If the goal behind generating this file is to be usable by AI systems, it is worth knowing where that goal is genuinely won and lost — because it is not here.

1. Can the agents fetch you? Confirmed in your access logs, not your robots.txt. A WAF rule quietly dropping unfamiliar user-agents is the single most common cause of a site being invisible to assistants, and it produces no error anybody notices.
2. Is the substance in the served HTML? Many retrieval fetchers parse rather than render. For a static site this is already solved — and it is the main reason SSGs do well here, far more than any text file at the root.
3. Does your markup describe the page accurately? Structured data disambiguates. It does not confer preference, and there is no verified evidence that denser schema wins citations.
4. Has anybody written an answer worth quoting? The only item with unbounded upside, and the only one no build step can generate for you.

Generate the file — it is cheap, and cheap things that might pay off later are worth having. Then go and do the fourth thing, which is the one that would still be worth doing if llms.txt vanished tomorrow.

Frequently Asked Questions