Fixing Sitemap and Robots.txt Drift After a Migration

What drift looks like after launch

"Drift" is the gap between what the two files say and what the new site actually is. It appears because both files are usually generated by the platform, copied from staging, or hand-written once and never revisited. The table lists the patterns that turn up most often after a move.

The Site Migration Audit's indexing and lost-URL checks often point here first: a sudden drop in crawled pages, or large numbers of "Blocked by robots.txt" and "Page with redirect" statuses, usually trace back to one of these rows.

Staging protection that survives launch

Staging sites are rightly hidden from search engines, but the protection can use several layers, and each one has to be removed separately. Check for all of these on the live site:

• robots.txt. A blanket Disallow: / under User-agent: *.
• Meta robots. A <meta name="robots" content="noindex"> in the template. In WordPress this comes from the "Discourage search engines from indexing this site" box under Settings, Reading.
• HTTP header. An X-Robots-Tag: noindex header added in the server or CDN configuration. It is invisible in the page source, so check the response headers.
• Password or IP restriction. Basic authentication or an IP allow-list left on some paths, such as the sitemap or media folder.
• Environment variables. Frameworks and hosts often switch robots rules based on an environment flag. If the production build was made with the staging flag, the wrong rules ship.

curl -s https://www.example.co.uk/robots.txt
curl -sI https://www.example.co.uk/ | grep -i "x-robots-tag"
curl -sI https://www.example.co.uk/sitemap.xml | grep -i "^HTTP\|x-robots-tag\|content-type"

Remember that robots.txt and noindex do different jobs. A robots.txt block stops crawling, so a search engine may never fetch the page and therefore never sees a noindex tag on it. If you need a page out of the index, it has to be crawlable so the noindex can be read.

Rebuilding the sitemap for the new site

The sitemap should be a list of the new site's canonical, indexable pages, and nothing else. The safest source is the new platform's own generator, because it updates as pages are published. A one-off export works for small static sites but drifts as soon as content changes.

What every listed URL must be

• An absolute address on the new host with the final scheme, such as https://www.example.co.uk/.
• Returning 200, not a redirect, a 404 or a login page.
• Self-canonical: the page's canonical tag points at the same address.
• Indexable: no noindex in the meta tags or headers, and not blocked by robots.txt.

Size and structure

A single sitemap file may hold up to 50,000 URLs and 50 MB uncompressed. Larger sites use a sitemap index that lists several files. Splitting by section, such as products, categories and articles, also makes the Search Console reports more useful, because you can see which section is being indexed slowly.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.co.uk/sitemaps/products.xml</loc>
    <lastmod>2026-09-26</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.co.uk/sitemaps/articles.xml</loc>
    <lastmod>2026-09-27</lastmod>
  </sitemap>
</sitemapindex>

lastmod after a migration

Google uses lastmod when it is consistently accurate and ignores priority and changefreq. A common migration mistake is stamping every page with the launch date, because the import touched every record. Carry the real last content change across from the old CMS if you can. If you cannot, leave lastmod out rather than filling it with a date that is not true.

Platform notes

The platform-specific guides for WordPress, Shopify and Next.js cover the settings in more detail outside a migration context.

The fix, step by step

The old domain: what to leave in place

When a domain changes, the instinct is to shut the old one down. For search engines, the old host still has an important job: every old URL must answer with a redirect for long enough that crawlers revisit it and transfer its signals.

• Do not block the old host in robots.txt. A disallowed URL is not fetched, so its redirect is never seen and the old address can linger in results.
• Keep the old Search Console property. It shows how quickly old URLs are being processed, and it is where the Change of Address tool is used for domain moves.
• Consider a temporary sitemap of old URLs. Listing old addresses, which now redirect, can prompt crawlers to revisit them sooner. It will show redirect warnings in Search Console, which are expected here. Remove it once most old URLs have dropped out of the index.
• Keep the old domain registered. Letting it lapse removes every redirect at once and hands its links to whoever buys it next.

Stopping drift on the next release

Both files are small enough to check automatically. A few lines in the deployment pipeline catch most drift before it reaches search engines.

# fail the deploy if the live robots.txt blocks everything
curl -s https://www.example.co.uk/robots.txt | grep -qx "Disallow: /" && { echo "robots.txt blocks the site"; exit 1; }

# fail if the sitemap mentions the staging host
curl -s https://www.example.co.uk/sitemap.xml | grep -q "staging.example.co.uk" && { echo "staging URLs in sitemap"; exit 1; }

• Keep robots.txt in version control, with separate files for staging and production chosen by the build rather than edited by hand.
• Protect staging with a password rather than relying only on robots.txt, so the robots file can stay identical between environments.
• Schedule a monthly crawl of the sitemap to catch redirects and 404s creeping back in as content changes.

The Robots and Sitemap Guide explains both files in full, beyond the migration case.