robots.txt unreachable, 404, 403 or 500: what Google does and how to fix it

Why the robots.txt response matters so much

robots.txt is a plain text file at the root of a host, such as https://www.example.com/robots.txt. Crawlers that follow the Robots Exclusion Protocol, Googlebot included, request it before crawling and keep a copy for a while. Google generally caches it for up to 24 hours, and longer if it cannot fetch a fresh copy.

The rules inside the file only matter if Google can read them. When it cannot, Google has to decide what to do with no instructions, and it bases that decision on the status code. A file that is simply absent is read as no restrictions. A server that seems to be failing is read as a reason to stay away, because Google cannot know whether the missing rules would have blocked anything.

Each combination of protocol, host and port has its own file. https://example.com/robots.txt, https://www.example.com/robots.txt and https://shop.example.com/robots.txt are three separate files, and a broken one only affects its own host.

What Google does with each response

Google documents its handling in its robots.txt specification. In summary:

If the file stays unreachable for more than 30 days, Google's documentation says it uses the last cached copy, or crawls without restrictions if it has no copy. That is not a safety net to rely on: a month of slow or stopped crawling means new pages are not found and changed pages are not refreshed.

Two details catch people out. First, a 403 on robots.txt is treated as no rules, which surprises owners who expected it to block crawlers. If you want to block crawling, return a 200 with Disallow rules. Second, Google only reads the first 500 KiB of the file and ignores anything after that, which can matter for very long generated files.

How to check the response

Request the file for every host you serve, including the bare domain and the www version, over both HTTP and HTTPS:

for u in http://example.com https://example.com http://www.example.com https://www.example.com; do
  echo "== $u"
  curl -sIL -o /dev/null -w "%{http_code} %{content_type} %{url_effective} %{time_total}s\n" "$u/robots.txt"
done

Each line should end in 200 text/plain at the address you expect, with a response time well under a second. Then repeat the request with a crawler user agent, because firewalls often treat it differently:

curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://www.example.com/robots.txt

A firewall may treat a real Googlebot request, coming from Google's own IP addresses, differently from your test, so the curl result is a clue rather than proof. For the real picture, look at your access log for requests to /robots.txt and their status codes:

sudo grep '"GET /robots.txt' /var/log/nginx/access.log | grep -i googlebot | awk '{print $4, $9}' | tail -n 20

What Search Console shows

The robots.txt report, under Settings, lists the robots.txt files Google found for the top hosts in your property, when each was last crawled, the fetch status and any warnings or errors in the rules. You can ask Google to recrawl a file from the report after a fix. The Crawl stats report, also under Settings, shows host status with a robots.txt fetch check, which highlights periods when the file could not be fetched. For problems with the rules themselves rather than the fetch, see fixing robots.txt syntax.

The usual causes of fetch failures

The common thread is that robots.txt depends on something more fragile than it needs to. A static text file served directly by the web server or CDN is about as reliable as anything on the internet. The same file built on every request by PHP, a database and a plugin fails whenever any of those fail. Our server error (5xx) guide covers what to do when pages are failing too, not just this one file.

Serving a reliable robots.txt

If you have nothing to block, a minimal file is enough. It allows all crawling and tells crawlers where your sitemap is:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Save it as UTF-8 plain text in the web root and let the web server serve it directly. In nginx, an exact-match location makes sure it never reaches the application:

location = /robots.txt {
    try_files $uri =404;
    default_type text/plain;
    access_log off;
}

If a CMS generates the file and you want to keep that, put a short cache in front of it so a brief outage does not turn into a fetch failure. On Apache, a real robots.txt file in the document root is served before WordPress rewrite rules run, because those rules only apply when no file exists at the requested path.

Letting crawlers past the firewall

In Cloudflare, create a custom rule that matches the path /robots.txt and uses the Skip action for the security features that challenge bots, or at least skips them for verified bots where cf.client.bot is true. Check security events afterwards to confirm crawler requests for the file are no longer challenged. The same idea applies to other CDNs and hosting firewalls: robots.txt should never sit behind a challenge page. The AI crawler access guide covers allowing specific bots through Cloudflare more broadly.

A worked example

A small online shop switches on a bot protection service after a wave of scraping. Within two weeks, Search Console's Crawl stats show crawl requests falling to almost nothing, and new products are not appearing in search. The pages themselves load fine in a browser.

The robots.txt report shows recent fetch failures, and the access log on the origin shows no Googlebot requests for /robots.txt at all. The protection service is answering them at the edge with a challenge page and a 503 status. Google reads that as a server error and stops crawling the site.

The owner adds a rule allowing verified crawlers to fetch /robots.txt and the sitemap without a challenge, then uses the robots.txt report to request a recrawl. The next fetch succeeds with a 200, and over the following days Crawl stats show requests climbing back to their usual level. The scraping protection stays on for everything else.

The fix, step by step

Edge cases worth knowing

• Pausing crawling on purpose. If you need Google to back off for a short time, such as during maintenance, Google recommends returning 503 for all URLs rather than blocking with robots.txt. Keep it short; a long run of errors slows crawling and can eventually affect indexing.
• Subdomains and CDNs. Images or files served from a separate host, such as cdn.example.com, are governed by that host's robots.txt. A failing file there can stop Google crawling your images.
• Redirect to a different host. Redirecting http://example.com/robots.txt to https://www.example.com/robots.txt is normal and works. Google follows the redirect and uses the rules it finds at the end.
• Other crawlers. Bing and other search engines have their own handling, which is broadly similar but not identical. A file that returns 200 quickly works for all of them.
• Staging copies. A staging robots.txt with Disallow: / copied to the live site is a rules problem, not a fetch problem, but it is common after launches. See fixing robots.txt blocks.

Keeping it fixed

• Add /robots.txt on each live host to your uptime monitor, checking for a 200 status and a known line of text such as your Sitemap line.
• Whenever you add or change a firewall, bot protection or rate limiting rule, request robots.txt with a crawler user agent and check the security events log for crawler requests.
• Prefer a static file. If your CMS must generate it, cache it.
• Look at the robots.txt report and Crawl stats in Search Console once a month, and after any hosting change.
• Keep the file short and readable. The robots and sitemap guide explains which rules are worth having and which are not.