Robots.txt tells Google which pages not to crawl. Your XML sitemap tells Google which pages to crawl. Getting either wrong can silently remove your pages from Google's index — here is how to check and fix both.
🤖 Test Robots & Sitemap All Audit Tools →The directive Disallow: / under User-agent: * blocks every crawler from crawling any page on your site. This is a catastrophic error that removes your entire site from Google. Always test robots.txt changes using the Robots & Sitemap Tester before deploying.
Google renders pages using a headless browser. If your robots.txt blocks the CSS and JavaScript files that build your page layout, Google sees a broken, unstyled page and may classify it as low quality. Allow all CSS, JS and font files.
A valid sitemap is an XML file listing your canonical, indexable URLs. Every URL in the sitemap should return HTTP 200, have a canonical tag pointing to itself, and not have a noindex robots directive. Include lastmod dates so Google knows when pages were last updated.
Robots.txt controls crawling. It does not control indexing. These are different things, and conflating them is the single most expensive misunderstanding in technical SEO.
Disallowing a URL in robots.txt tells Google not to fetch it. It does not tell Google not to list it. If other pages link to that URL, Google may index it anyway — showing it in results with no title and a description reading that no information is available, because it was forbidden from looking. The page is in the index and Google has no idea what is on it.
Worse, and more common: a page that is disallowed in robots.txt and carries a noindex tag will never be removed from the index, because Google cannot fetch the page to read the tag telling it to go away. The two directives cancel each other out, and the page stays.
noindex. To save crawl budget on pages you do not care about, disallow it. Never both — the combination is self-defeating and it is one of the most common configuration errors on the web.Two characters that remove an entire site from search. It happens when a staging configuration is deployed to production, and it is the fastest way to lose all organic traffic. Check it after every deploy — not because it is likely, but because the cost of missing it is total.
Disallowing /assets/, /js/ or /css/ is a habit inherited from an era when it seemed tidy. Today it prevents Google from rendering your page: it can fetch the HTML but not the code and styles that fill it, so it renders a broken shell and judges the page on that. Google needs your resources. Let it have them.
Robots.txt is a public file listing the paths you would prefer people did not visit. It is read by anyone curious, and it is obeyed only by crawlers that choose to. Putting /admin/ or /private-backups/ in it is not protection — it is a signpost. Anything genuinely sensitive requires authentication, not a polite request.
Google applies the most specific matching rule, not the first one. A broad Disallow followed by a narrow Allow does permit the narrower path — but the behaviour differs between crawlers, and complicated rule sets are where mistakes hide. Keep the file short and obvious.
A crawler reads only the block that matches it most specifically. If you have a block for Googlebot and a block for *, Googlebot reads only its own block and ignores the wildcard entirely — including any rules you assumed were universal.
A sitemap is a discovery aid. It tells Google which URLs you consider worth crawling, and it is particularly useful for pages that are poorly linked internally, newly published, or otherwise hard to find.
It is not a guarantee of indexing. Submitting a URL in a sitemap does not oblige Google to crawl it, and certainly does not oblige it to index it. A site with a thousand sitemap entries and two hundred indexed pages does not have a sitemap problem — it has a quality problem, and the sitemap is faithfully reporting it.
Nor is a sitemap a substitute for internal linking. A page that appears in the sitemap and is linked from nowhere is an orphan, and orphans are routinely crawled once and never returned to. Google reads internal links as a statement about what you consider important; a page nothing links to is a page you have implicitly declared unimportant, whatever the sitemap says.
lastmod that updates on every build, for every page, whether or not anything changed, is noise. Google learns to ignore it. An accurate lastmod is genuinely useful; a fabricated one is worse than none.Sitemap: line with the full URL is how crawlers other than Google find it without being told.This point deserves its own section because it causes real damage during migrations and audits.
A sitemap contains the URLs you knew about and chose to declare. It systematically omits the ones that matter most in a crisis: the old post from years ago that quietly accumulated links, the discontinued product page that still ranks for a term nobody remembers targeting, the orphaned landing page from a campaign that ended. None of these are in your sitemap. Several of them are earning.
Any process that starts from the sitemap — building a redirect map, auditing content, deciding what to keep — will therefore miss exactly the pages whose loss hurts most. The complete picture requires a crawl, a Search Console export of pages that received impressions, and ideally the server logs. The gap between that union and your sitemap is precisely the traffic that gets lost in a migration.
Crawl budget is discussed far more often than it applies. For a site of a few hundred pages, it is essentially irrelevant — Google will crawl everything it wants to, and the constraint is never the budget.
It becomes real on large sites, and its symptom is specific: pages that Google has discovered and not crawled. If Search Console reports a large number of URLs as "Discovered — currently not indexed", Google has looked at your site and decided the remainder is not worth fetching. That is a judgement about quality and authority, and it is the honest signal underneath most crawl budget conversations.
What genuinely wastes it:
The last of these is the one nobody wants to hear. Publishing more pages does not earn more crawling. It spreads the same crawling more thinly, and if the pages are thin, it lowers Google's estimate of what the rest of the site is worth.
A single sitemap containing every URL is a missed opportunity. Split by type, and the sitemap stops being a submission and becomes a diagnostic.
Search Console reports indexing coverage per sitemap. If products, articles and category pages sit in one file, you learn that 60% of the site is indexed and nothing about which 40% is not. Split them — products, articles, categories, each in its own file behind a sitemap index — and the same report tells you that products are indexed at 95% and articles at 20%, which is an entirely different and immediately actionable fact.
This costs nothing to implement and it is the difference between knowing you have a problem and knowing where it is.
Do not split arbitrarily by number. A sitemap containing URLs 1–50,000 and another with 50,001–100,000 tells you nothing at all, and that is how most large sitemaps are split.
The instinct on discovering unindexed pages is to submit the sitemap again. This almost never helps, because resubmission is not the constraint — Google already knows the URLs exist. Work through the actual causes in order.
Check robots.txt and the noindex tag, including the X-Robots-Tag HTTP header, which is invisible in the page source and is a common cause of pages disappearing after a server configuration change. All three must be clear.
A page linked from nowhere is an orphan. The sitemap will get it discovered; it will not get it valued. Internal links are how you tell Google a page matters, and a page with no inbound links has been implicitly declared unimportant by its own site.
If Google has decided the page is a near-duplicate of another, it will index one and drop the other, and Search Console will report it as an alternate with a proper canonical or a duplicate without a user-selected canonical. That is not a bug to be argued with; it is a judgement about the content.
The uncomfortable answer, and the correct one more often than any of the above. "Discovered — currently not indexed" means Google found the URL and chose not to spend a crawl on it. That is a verdict on the site's quality and authority, and no amount of sitemap work overturns it. The remedy is fewer, better pages — not more submissions.
No. It stops Google crawling the page, not indexing it. A disallowed page that other sites link to can still appear in results, with no title and no description, because Google was forbidden from looking at it. To keep a page out of the index, allow the crawl and serve noindex.
You can, and it is self-defeating. Google cannot fetch the page to read the noindex, so the page is never removed. Use one or the other, never both.
No. Google needs them to render the page. Blocking them means it renders a broken shell and judges the page on that.
No. A sitemap aids discovery; it obliges nothing. If a large share of your sitemap is not indexed, the sitemap is not the problem — it is faithfully reporting a quality or authority problem.
No. Google has stated it ignores both. They are harmless and inert.
Never. It is a public file that lists the paths you would rather people did not visit, and it is obeyed only by crawlers that choose to obey it. Anything sensitive requires authentication.
Run the Robots & Sitemap Tester and get actionable results in minutes. Pay as you go — no subscription needed.
Test Robots & Sitemap →aiwebpageseo.com is a data-driven SEO and AEO (Answer Engine Optimisation) platform providing a free suite of technical website tools. Rather than relying on AI-theorised assumptions, the platform analyses live URL performance, delivering objective diagnostics, page speed metrics, CLS debugging, and site crawl data alongside actionable technical tutorials.