Robots.txt and Sitemap Fixes: Getting the Crawl Rules Right
Two small files shape how search engines treat your whole site. robots.txt says which
paths crawlers may request, and the XML sitemap lists the pages you want them to know about. Because
they are short and rarely looked at, mistakes in them can sit unnoticed while whole sections drop out
of search.
The Robots & Sitemap Tester fetches your robots.txt, checks its syntax, identifies what
it blocks, fetches every declared sitemap, validates the XML, checks each listed URL answers with a 200,
flags inconsistent lastmod dates and looks for crawler-trap risks. This page explains each
finding and its fix.
Two files, two different jobs
Much of the confusion around these files comes from expecting them to do things they do not do. It is worth being precise.
| robots.txt | XML sitemap | |
|---|---|---|
| What it does | Tells compliant crawlers which URL paths they may not fetch. | Lists URLs you want search engines to discover, with optional last-modified dates. |
| What it does not do | It does not remove pages from the index, and it does not hide anything from people or badly behaved bots. | It does not force indexing, and listing a page does not make it rank. |
| Where it lives | Always at the root of the host: https://www.example.co.uk/robots.txt.
Each host and protocol has its own. | Anywhere on the host, declared in robots.txt and submitted in Search Console. |
| When it goes wrong | Pages you want in search stop being crawled. | Search engines waste time on dead URLs and lose trust in the dates you give. |
The key point is that robots.txt controls crawling, not indexing. If you want a page kept out of search
results, it needs a noindex meta tag or header that crawlers are allowed to fetch and read.
Blocking it in robots.txt can have the opposite effect, which is the subject of one of the guides below.
The fix guides
The finding guides are grouped by which file the problem lives in, followed by platform guides.
Problems in robots.txt
Fixing accidental robots.txt blocks
covers the full-site Disallow: / left over from staging and broad patterns that catch public
pages by mistake, with a pre-launch check to prevent it. Fixing
robots.txt syntax errors covers misplaced wildcards, malformed User-agent lines and
directives search engines silently ignore. Fixing
a missing or wrong sitemap declaration covers the Sitemap: line, which must be a full
absolute URL pointing at the XML file or sitemap index.
Problems in the sitemap
Fixing dead URLs in the sitemap covers
removing errors, redirects and noindexed pages so the sitemap lists only live, indexable URLs. Fixing
oversized sitemap files covers the limit of 50,000 URLs or 50 MB uncompressed per file and splitting
into a sitemap index by section, type or date. Fixing
lastmod and priority misuse covers keeping lastmod honest and why priority
values are not worth fussing over.
Problems between the two
Fixing crawler-trap patterns covers endless filter combinations, calendars that go back for decades and other URL spaces that never end. Fixing conflicting noindex and Disallow covers the common mistake of blocking a page and noindexing it at the same time, so the noindex is never seen.
By platform
The WordPress guide covers
sitemaps generated by core or by Yoast and Rank Math, virtual and physical robots.txt files and multisite.
The Shopify guide covers the
automatic sitemap index and the limited ways robots.txt can be customised. The Next.js
guide covers generating both files from app/robots.ts and app/sitemap.ts
and handling very large sites at build time.
Which findings to fix first
- Anything that blocks the whole site or a major section. A single
Disallow: /underUser-agent: *stops compliant crawlers fetching anything. Fix this within the hour, not the week. - A robots.txt that returns a server error. Google has documented that if robots.txt returns a 5xx error it may treat the site as fully blocked for a time. A missing robots.txt that returns a 404 is treated as no restrictions, which is far less harmful.
- Noindex and Disallow on the same URL. The page you meant to remove may stay in results.
- Syntax errors. A broken rule is usually ignored, so it may be allowing or blocking something you did not intend.
- Dead or redirected URLs in the sitemap. They waste requests and make the sitemap look unreliable.
- Missing or relative sitemap declaration. Easy to fix, and it helps every search engine find the sitemap, not only those you submit to by hand.
- Crawler traps. Serious on large or filtered sites; less urgent on small brochure sites.
- Oversized files and lastmod issues. Tidy these once the rest is clean.
How robots.txt rules are actually read
Many syntax findings come from assumptions about how rules combine. Google documents the following behaviour, and other major search engines work in a similar way:
- Groups: rules are grouped under one or more
User-agentlines. A crawler uses the most specific group that names it and ignores the others, so a crawler with its own group does not also follow the*group. - Longest match wins: when an
Allowand aDisallowboth match a URL, the rule with the longer path applies. If they are equally long, Google uses the less restrictive one. - Wildcards:
*matches any sequence of characters and$marks the end of the URL. Paths are case-sensitive. - Unsupported lines are ignored: Google does not support
noindexin robots.txt, and ignorescrawl-delay. - Size: Google reads up to 500 KiB of the file. Anything after that is ignored.
User-agent: *
Disallow: /basket/
Disallow: /*?sort=
Allow: /basket/help
Sitemap: https://www.example.co.uk/sitemap.xmlIn this example every crawler is kept out of basket pages except the help page, sorted listing URLs are blocked, and the sitemap is declared with its full address. Search Console's robots.txt report shows the version Google last fetched and any lines it could not parse.
Keeping the sitemap honest
A sitemap works best when search engines can trust it. If half its URLs redirect, return errors or
carry a noindex, it becomes a list of suggestions they learn to check rather than rely on.
The tester checks each listed URL for exactly this reason.
A trustworthy sitemap follows a few rules:
- Only final URLs: the canonical, HTTPS, correct-host version of each page, answering with a 200.
- Only indexable URLs: nothing blocked by robots.txt, nothing with
noindex, nothing whose canonical points elsewhere. - Real dates:
lastmodchanges when the main content changes, not every time the file is rebuilt or a sidebar updates. Google has said it useslastmodwhen it is consistently accurate. - Generated, not hand-written: a sitemap produced by the CMS or build process from the live pages stays in step with the site. A hand-edited file drifts out of date.
- Split sensibly: on larger sites, separate sitemaps for products, articles and categories make it easier to see which section has indexing problems in Search Console.
For how the sitemap compares with the pages a crawler actually finds through links, see the Site Crawler Guide.
Order of work
- Run the Robots & Sitemap Tester and save the current robots.txt as a backup.
- Remove any rule that blocks pages you want in search, and make sure robots.txt itself returns a 200 or a 404, never a server error.
- Correct syntax errors and simplify the file; fewer, clearer rules are easier to reason about.
- For pages you want removed from results, lift any Disallow and use
noindexinstead until they drop out. - Add an absolute
Sitemap:line for your sitemap or sitemap index. - Regenerate the sitemap so it lists only live, indexable, canonical URLs with honest dates, split into an index if needed.
- Block genuine crawler traps with targeted patterns, checking that no real content is caught.
- Submit the sitemap in Search Console, check its robots.txt report, and run the tester again.
The example report shows the tester's output on a real configuration, and the Robots & Sitemap Guide is the full reference. For wider on-page checks, see the Site Audit Guide.
1. Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops compliant crawlers from fetching the page, but Google can still index the URL if other pages link to it, usually showing it with no description. To remove a page from results, allow it to be crawled and add a noindex meta tag or X-Robots-Tag header, then wait for search engines to recrawl it. Once it has dropped out you can decide whether to block crawling as well, though usually there is no need.
2. Do I need a robots.txt file at all?
No. If a site has no robots.txt and the URL returns a 404, search engines treat that as permission to crawl everything, which is fine for many small sites. A robots.txt becomes useful when you want to keep crawlers out of areas such as basket pages, internal search results or filter combinations, or to declare your sitemap location. What you must avoid is a robots.txt URL that returns a server error, because that can be treated as a block.
3. Where should my sitemap be and what should it be called?
It can be at any URL on the same host, and the name does not matter to search engines, though sitemap.xml or sitemap_index.xml at the root are common and easy to find. What matters is that it is valid XML, returns a 200 status, and is declared with a full absolute URL on a Sitemap line in robots.txt. Submitting it in Google Search Console and Bing Webmaster Tools as well gives you reports on how it is being processed.
4. Does Google use the priority and changefreq values in my sitemap?
Google has said it ignores both priority and changefreq. It does use lastmod, but only when the dates are consistently accurate, meaning they change when the page content genuinely changes. Other search engines may treat these fields differently, so there is no harm in including them if your system generates sensible values. Just do not spend time tuning them, and never set every lastmod to today's date on each rebuild, as that teaches search engines to ignore it.
5. How many URLs can one sitemap contain?
A single sitemap file can list up to 50,000 URLs and must be no larger than 50 MB uncompressed. Larger sites split their URLs across several sitemap files and list those files in a sitemap index, which can itself reference up to 50,000 sitemaps. Splitting by content type, such as products, categories and articles, also makes it easier to see in Search Console which section has pages that are not being indexed.
6. Why is Google indexing a page I blocked in robots.txt?
Because the block prevents crawling, not indexing. If other pages or other websites link to the blocked URL, Google may still list it, usually with a note that no information is available. Google cannot see any noindex tag on the page because it is not allowed to fetch it. Remove the Disallow for that URL, add noindex, and let it be recrawled. Search Console reports this situation as indexed though blocked by robots.txt.
7. Should I block AI crawlers in robots.txt?
That is a business decision rather than a technical fix, and it is separate from search engine indexing. Several AI companies publish crawler names that respect robots.txt, and some separate crawlers used for training from those used to fetch pages when answering a question. Blocking the latter can remove you from AI answers. If you do add rules, name each crawler explicitly in its own group, and check that the rules do not also catch Googlebot or Bingbot.
The Robots & Sitemap Tester fetches your robots.txt and every declared sitemap, validates both, checks each listed URL responds with a 200, flags inconsistent lastmod dates and tests for crawler-trap risk.
Run the Robots & Sitemap Tester →