Robots and Sitemap Example: Disallow Rules and XML Audit

This is the most consequential misunderstanding in technical SEO, and it produces a failure that looks impossible until you know the mechanism.

A Disallow rule tells a crawler not to fetch the URL. It does not tell Google not to index it. If Google discovers the URL another way — an external link, an internal link, a sitemap — it can index it without ever fetching it. The result is a search listing for a page Google has never read: the URL, no title it can trust, and often the text "No information is available for this page."

The rule of thumb: robots.txt is for controlling crawl budget. noindex is for controlling the index. They are different tools for different jobs and using one for the other's purpose fails silently.

The score is docked to 70% on "No Blocks" because three Disallow rules exist. But /admin/, /private/ and /wp-login.php are exactly what should be blocked. The report is measuring the presence of rules, not their correctness, and a robots.txt with zero disallow rules is not a better one — it is usually one nobody has configured.

Judge the rules, not the count. What actually makes a robots.txt dangerous:

• Blocking CSS or JavaScript. Google renders pages. Deny it the stylesheet and it sees an unstyled document; deny it the JavaScript on a client-rendered site and it sees an empty one. This is the single most damaging robots.txt mistake and it is still common in old configurations that blocked /assets/ or /wp-includes/ wholesale.
• Disallow: / left over from staging. Two characters. It takes the entire site out of the index. Check for it after every migration.
• A rule broader than intended. Disallow: /blog blocks /blog-archive and /blogging-tips too — there is no implicit boundary. Write Disallow: /blog/ if you mean the directory.
• Blocking a URL you also need de-indexed. See above. It is the one that keeps the page in.

"3 AI crawlers permitted" is a green tick, and the underlying picture is more granular than a single decision. Different bots from the same company do different jobs, and blocking the wrong one has consequences people do not intend:

Two of these rows correct widespread errors. First: blocking GPTBot does not remove you from ChatGPT — training and search are separate crawlers, and a publisher who blocks GPTBot to protect their content from training while hoping to be cited in ChatGPT's search results has done precisely what they intended, so long as they left OAI-SearchBot alone.

Second: Google-Extended has no effect on Google Search. It governs whether your content is used for Gemini training. Blocking it does not remove you from the index, does not affect rankings, and does not remove you from AI Overviews — those are served by Googlebot, and opting out of them means opting out of Search entirely. That trade-off is the real decision, and it is not made in this file.

A sitemap is a discovery aid. It tells Google which URLs exist. It does not request indexing, it does not promise indexing, and submitting it does not cause indexing.

This is why the most common sitemap disappointment is not a sitemap problem at all: 1,204 URLs submitted, and a Search Console report full of "Discovered — currently not indexed". That status means the sitemap worked perfectly. Google found the URLs, looked at them, and declined. The sitemap did its job and the content did not.

What a sitemap must contain, and what audits rarely check:

• Only URLs that return 200. A redirecting URL in a sitemap is a contradiction — the file says "this is a canonical page", the server says "it moved".
• Only canonical URLs. A URL that canonicalises somewhere else has no business being listed.
• Nothing carrying noindex. You are asking Google to look at a page you have told it to ignore.
• Nothing blocked in robots.txt. Same contradiction, in the two files Google trusts most.

The limits: 50,000 URLs and 50MB uncompressed per file; beyond that, use a sitemap index. And a sitemap index is worth using well below the limit anyway — split by section (products, blog, categories) and Search Console will report indexing coverage per sitemap, which turns "660 pages not indexed" into "the entire product template is not indexed". That is a diagnosis rather than a number.

The report notes lastmod on 1,198 of 1,204 URLs. The six without are not the problem worth thinking about. This one is:

changefreq and priority: Google ignores both. They have been ignored for years. priority in particular is universally set to 1.0 by plugins and conveys nothing — you cannot tell Google which of your pages matter by asserting that all of them do. If you want to signal that a page is important, link to it more, from pages that already have authority. That is the only version of this signal that Google reads.

And a mechanical point that catches people out: robots.txt is per-origin. The file at https://example.com/robots.txt governs only that protocol and that host. A subdomain needs its own file at its own root, and rules placed in the main site's robots.txt have no effect on it whatsoever.

Should I block AI bots in robots.txt?

Only if you have a specific reason to prevent AI models from training on or citing your content — and note that the bot names are not interchangeable. GPTBot is OpenAI's training crawler, so blocking it stops your content being used for training but does not remove you from ChatGPT's search citations, which come from a separate crawler, OAI-SearchBot. Blocking PerplexityBot prevents Perplexity citations. Google-Extended governs Gemini training only and has no effect whatsoever on Google Search. For most sites seeking visibility in AI search, allowing the search and citation crawlers is beneficial — they are potential sources of citation and traffic.

What is the ideal XML sitemap structure?

A valid XML sitemap contains loc (URL), lastmod (last modified date in ISO 8601 format) and optionally changefreq and priority. It must be under 50MB and 50,000 URLs. For large sites, use a sitemap index file. Submit your sitemap to Google Search Console after any significant changes.

What causes a robots.txt score below 90?

Common issues: no sitemap reference, overly broad disallow rules that block important pages, inconsistent user-agent casing, incorrect syntax (missing User-agent before Disallow), blocking CSS or JavaScript files Google needs to render pages, or blocking AI crawlers unintentionally.

Does Disallow in robots.txt remove a page from Google?

No. Disallow stops the crawl, not the indexing. If Google discovers the URL another way — an external link, an internal link, a sitemap — it can index it without ever fetching it, producing a listing with no usable title and often the text "No information is available for this page". Worse, adding both Disallow and noindex cancels them out: the noindex lives inside the page, so blocking the crawl means Google never reads it. To remove a page, allow the crawl, serve noindex, wait for the recrawl, and only then consider a Disallow.

Does blocking Google-Extended affect my Google Search rankings?

No. Google-Extended controls only whether your content is used for Gemini model training. It does not remove you from the index, does not affect rankings, and does not remove you from AI Overviews — those are served by Googlebot, so opting out of them would mean opting out of Search entirely. Similarly, blocking GPTBot stops OpenAI using your content for training but does not remove you from ChatGPT's search citations, which come from a separate crawler, OAI-SearchBot.

Are changefreq and priority worth setting?

No — Google ignores both, and has for years. Priority in particular is set to 1.0 by most plugins and conveys nothing; you cannot tell Google which pages matter by asserting that all of them do. To signal that a page is important, link to it more from pages that already have authority. That is the only version of this signal Google reads.

My sitemap has 1,204 URLs but most are not indexed. Is the sitemap broken?

Probably not. A sitemap is a discovery aid — it tells Google which URLs exist. It does not request or guarantee indexing. A report full of "Discovered — currently not indexed" means the sitemap worked perfectly: Google found the URLs, assessed them, and declined. The sitemap did its job and the content did not. Do check that every listed URL returns 200, is canonical, carries no noindex, and is not blocked in robots.txt — but if all that holds, the problem is the pages themselves.