llms.txt Not Found: Why the File Is Missing and What to Do
The auditor reports llms.txt as missing when a request for /llms.txt on your domain does not come back with a text file. Sometimes there really is no file. Just as often the file exists but the server answers with an error page, the home page, a redirect or a firewall challenge instead.
This guide shows you how to tell those cases apart in a couple of minutes, gives a short starter file you can publish today, and explains the server, host and CDN settings that stop a file you uploaded from being found.
What "missing" means to the auditor
A tool that looks for llms.txt has one place to look: the root of the host, at https://www.example.co.uk/llms.txt.
It has no way to discover a file elsewhere unless something links to it. So the question is not whether
the file exists somewhere on your server, but what happens when that one address is requested by a program
rather than a person with a browser.
The auditor treats the file as missing whenever that request does not end in a usable text file. The common results look like this:
| What comes back | What it usually means | Where to fix it |
|---|---|---|
| 404 Not Found | No file at the root, or it is in a subfolder, or the name
differs in case (LLMS.txt, llms.TXT) on a case-sensitive Linux server | Upload or rename the file |
| 200 with an HTML page (a soft 404) | A CMS or JavaScript app answers every unknown address with its own page | Web server or app routing |
| 301 or 302 to the home page | A redirect plugin or rule sends all missing pages home | Redirect settings |
| 301 to another host that then fails | The file is on the www host but not the bare domain, or on the old domain only | Host names and redirects |
| 403 Forbidden or a challenge page | A firewall, CDN bot rule or security plugin blocks non-browser clients, or file permissions are wrong | Firewall, CDN or permissions |
| 5xx error or a timeout | The request is being passed to an application that fails on it | Server configuration |
Each row has a different fix, which is why the first step is always to look at the raw response rather than opening the address in a browser. A browser hides redirects, runs JavaScript and passes challenges, so it may show you a file that a crawler never gets.
Diagnosing it in two minutes with curl
Run these commands in a terminal (macOS and Linux have curl built in; recent Windows versions do too). Replace the domain with your own.
curl -sS -o /dev/null -w "%{http_code} %{content_type} %{redirect_url}\n" https://www.example.co.uk/llms.txt
curl -sS -o /dev/null -w "%{http_code} %{content_type} %{redirect_url}\n" https://example.co.uk/llms.txt
curl -sSL -A "Mozilla/5.0 (compatible; test-bot/1.0)" https://www.example.co.uk/llms.txt | head -n 5The first two lines print the status, content type and any redirect target for the www and bare host. The third follows redirects with a bot-like user agent and shows the first five lines of whatever arrives. Read the results like this:
200 text/plainand a first line starting with#: the file is found. If the auditor still reports it missing, re-run it; if it persists, the auditor may be reaching a different host name.200 text/html: a soft 404. Look at the body; it will be your theme or app shell.301or302with a redirect URL: follow it and check where it ends. A redirect from the bare host to www is fine if the www copy returns the file.403, or a body mentioning a challenge, JavaScript check or access denied: a firewall or bot rule is in the way.404: the file is not where the server looks.
For comparison, run the same command against /robots.txt. If robots.txt returns a clean
200 text/plain and llms.txt does not, put llms.txt in the same folder as robots.txt and
the server will almost certainly treat them the same way. The one exception is a platform that generates
robots.txt itself rather than serving a file, which is how WordPress and Shopify behave out of the box.
A starter file you can publish today
An empty or placeholder file helps nobody, but you do not need a perfect one to get started. This is a complete, valid file for a fictional plumbing firm. Copy the shape, change every line, and keep it to the pages you would show a new customer first.
# Ashgrove Plumbing & Heating
> Gas Safe registered plumbing and heating engineers covering Sheffield
> and north Derbyshire. Domestic customers only; no commercial work.
Quotes are free and fixed-price for boiler installations. Emergency call-outs
are available 7am to 10pm.
## Services
- [Boiler installation](https://www.example.co.uk/boilers/): brands fitted, survey process, typical timescales
- [Boiler servicing and repair](https://www.example.co.uk/servicing/): annual service, common faults, call-out charges
- [Bathrooms](https://www.example.co.uk/bathrooms/): full refits, showers, what is included in a quote
## Customers
- [Areas covered](https://www.example.co.uk/areas/): postcodes served and travel charges
- [Contact and booking](https://www.example.co.uk/contact/): phone hours, online booking, emergency line
- [Guarantees](https://www.example.co.uk/guarantees/): workmanship guarantee and manufacturer warranties
That is enough to clear the missing finding. The structure rules are explained in the format
guide, and choosing and describing pages well is covered in the content
guide. Save the file as UTF-8 plain text named llms.txt, all lower case.
The fix, step by step
Step 1
Check what the address returns now
Before uploading anything, run the curl commands from the diagnosis section against both the www and the bare version of your domain and write down the status code, content type and any redirect for each. This tells you whether you are dealing with a file that genuinely is not there, or with a server, redirect or firewall problem. Uploading a file will not fix a soft 404 or a firewall block, so knowing which case you have saves time.
Step 2
Write a short starter file
In
a plain text editor, write an H1 with your business name, a two or three line summary in a blockquote,
and one or two H2 sections listing the six to fifteen pages that best explain what you do, each as a
Markdown link with a short note. Use full addresses on your main host name. Save it as llms.txt
in lower case, encoded as UTF-8, and keep a copy on your computer as the master version you will edit
later.
Step 3
Upload it to the document root
Put
the file in the same folder that holds robots.txt, favicon.ico or your home
page's index file. On Plesk that is usually httpdocs, on cPanel public_html,
and on many Linux servers /var/www/html or a site folder under /var/www. Make
sure the web server can read it, which normally means permissions of 644. Then request /llms.txt
with curl again and check for a 200 status.
Step 4
Stop the app or CMS from swallowing the request
If curl still shows your theme or app shell with a 200 status, the server
is passing the request to the application before looking for a real file. Add a rule that serves /llms.txt
directly from disk, using the Apache or nginx examples in this guide, or place the file in the framework's
public or static folder. Also check redirect plugins and rules that send every missing address to the
home page, and exclude this path from them.
Step 5
Let non-browser clients through the firewall
If you get a 403 or a challenge page, look in your CDN, hosting firewall
or security plugin for bot protection, JavaScript challenges or rules that block unknown user agents.
Add an exception that allows plain requests to the exact path /llms.txt, alongside the one
most sites already have for /robots.txt. Keep the exception narrow, limited to that path
and to GET and HEAD requests, so the rest of your protection stays in place.
Step 6
Confirm the host name and cache
Check
that the host your canonical URLs use serves the file, and that the other host redirects to it in a single
step rather than to the home page. Purge the CDN or page cache for /llms.txt, because a
cached 404 can survive for hours after the upload. Finally run the llms.txt Auditor again on the canonical
domain and confirm that the file is found and that the format and link checks now have something to report
on.
Server rules that make the file reachable
On most shared hosting a file placed in the document root is served straight away, because the standard rewrite rules only hand a request to the CMS when no real file matches. Problems start with custom rules, reverse proxies and JavaScript apps. Pick the block for your server.
Apache (.htaccess)
# Serve llms.txt as UTF-8 text and keep it out of the CMS rewrite
<Files "llms.txt">
ForceType text/plain
</Files>
AddCharset utf-8 .txt
RewriteEngine On
RewriteRule ^llms\.txt$ - [L]The RewriteRule line with a dash means "do nothing and stop", and it must sit above any
rule that sends everything to index.php. Most CMS rules already include RewriteCond %{REQUEST_FILENAME} !-f,
which skips real files; if yours lacks it, that is the cause.
nginx
location = /llms.txt {
charset utf-8;
try_files $uri =404;
}An exact-match location wins over the general location / block, so the request
never reaches try_files $uri /index.html or a proxy_pass to your app. The block
inherits the root of the server block, and the standard mime.types file already
maps .txt to text/plain; the charset line adds the encoding. Test
with nginx -t before reloading. On Plesk, add the block under the domain's Apache and nginx
settings as additional nginx directives.
JavaScript frameworks and static hosts
Next.js, Nuxt, Astro, Vite and similar tools copy anything in their public folder to
the site root as-is, so public/llms.txt becomes /llms.txt. Hugo uses static,
and Jekyll copies files from the project root unless they are excluded. If the site is a single-page
app behind a catch-all rewrite on Netlify, Vercel or similar, real files are normally served before rewrites
apply, but a forced rewrite rule can override that; check the rule list for anything marked as forced.
Hosted platforms and when uploading is not possible
Some platforms do not give you a document root at all.
- WordPress sites on normal hosting can take a file in the root folder by FTP or the host's file manager, and several SEO plugins can now generate the file. Details, including multisite, are in the WordPress guide. On WordPress.com, uploading to the root depends on your plan.
- Shopify does not let you upload files to the root, but since 2026 it serves
/llms.txton stores itself and lets a theme replace the content with a template. How that works is in the Shopify guide. - Website builders such as Wix, Squarespace and similar hosted builders control which
root files exist. Check the builder's current help pages for llms.txt or custom root file support, as
these features change. If there is no option, a page at another address can hold the content, but tools
looking at
/llms.txtwill not find it, and the auditor will keep reporting the file as missing.
If you cannot serve the file at the root, do not add a redirect from /llms.txt to an
ordinary HTML page and consider it done. Some tools follow redirects and some do not, and an HTML page
does not meet the format. Where the platform offers no route to a real text file, accept the finding
and put the effort into the pages themselves.
Things that look fixed but are not
- Testing only in a browser. A browser that has already passed a CDN challenge will keep showing the file because of a cookie, while a crawler gets a 403.
- Uploading to the wrong site. On servers hosting several domains, it is easy to drop the file in the folder of a different site or a subdomain. Check the path against the domain's document root setting in the control panel.
- A staging copy. A file uploaded to a staging site will vanish at the next deploy from staging to live, or vice versa. Add it to whatever source the live site is built from.
- A file blocked by robots.txt. A
Disallow: /*.txt$rule, occasionally added to hide text files, also covers llms.txt for crawlers that obey robots.txt. Read the robots.txt file for rules that match the path, and see the robots.txt AI blocks guide for AI crawler rules. - A deploy that wipes the root. Some deploy tools replace the whole document root on each release. If llms.txt is not part of the repository or build output, it disappears at the next release.
1. Do I need an llms.txt file at all?
No. It is optional, and no major AI search or answer provider has confirmed that it reads the file when choosing sources. Your pages are crawled and understood without it. The case for adding one is that it is cheap and some tools, especially developer and research tools, do look for it. If your site is a small brochure site and your time is short, fixing crawl access and page content will matter more than this file.
2. Why does the file open in my browser but the auditor says it is missing?
Your browser may be seeing something different from what a program sees. It could be passing a firewall challenge silently, using a cached copy, or following a redirect to a host that serves the file while the auditor's host does not. Run the curl commands in this guide with a non-browser user agent and compare the status codes. The difference nearly always points to a firewall rule, a cache or a host name mismatch.
3. Can I put llms.txt in a subfolder such as /docs/llms.txt?
You can, and the proposal allows files in subpaths for sections of a site, but tools and the auditor look at the root first. A file only in a subfolder will usually go unnoticed. If your documentation lives under a subfolder, publish a root llms.txt that describes the site and links to the subfolder file in one of its sections, so tools can find their way from the standard location to the detailed one.
4. What permissions should the file have?
On Linux hosting, 644 is normal: the owner can read and write, and everyone else, including the web server, can read. If you see a 403 status and there is no firewall involved, check that the file is readable by the web server user and that the folder above it allows access. Avoid 777 or other world-writable settings. They do not fix reading problems and they create a real security risk on shared servers.
5. Is a redirect from /llms.txt to another address acceptable?
A redirect to another copy of the same text file, for example from the bare domain to the www host, is fine as long as it is a single hop to a 200 response. A redirect to the home page, a login page or an HTML page that merely contains the text is not. Tools may or may not follow redirects at all, so the most reliable arrangement is the canonical host serving the file directly with no redirect.
6. Should I create llms-full.txt at the same time?
Not necessarily. The companion llms-full.txt file holds the full text of your key pages, and it is most useful for documentation sites with a lot of reference material. For a small business site, getting a sound llms.txt in place comes first. Once it passes the auditor, read the llms-full.txt guide and decide whether your content suits it. Adding it later is simple if you generate it from the same page list.
7. Will the missing finding affect my SEO score or Google rankings?
Google has not said it uses llms.txt, so a missing file does not affect Google rankings, and it is not something Search Console reports. The auditor flags it because the auditor's job is to check this file specifically. Treat it as a low-risk improvement for the tools that read it. If your crawl, indexing or Core Web Vitals reports show problems, those deserve your attention before this does, because they affect every visitor and every crawler.
The llms.txt Auditor fetches your /llms.txt and /llms-full.txt files, validates them against the proposed format, checks that each linked URL responds, confirms robots.txt allows the relevant AI crawlers and scores the quality of the content.
Open llms.txt Fixes →