Does Disallow in robots.txt deindex pages?

No. Disallow blocks crawl but not indexing. Pages already known to Google can stay indexed even after a Disallow is added — Google just stops re-crawling them. To deindex, use noindex meta or X-Robots-Tag HTTP header, NOT robots.txt. To deindex AND block crawl, use noindex first, wait for Google to process, then add Disallow.

Why is my robots.txt blocking everything?

Most common cause: staging environment has Disallow: / to keep crawlers out, and the staging config leaked into production during deployment. Other causes: copy-paste from someone else's robots.txt without understanding what it does, CMS plugin that auto-generates robots.txt with restrictive defaults, .htaccess rule serving the wrong file.

Should I block /wp-admin/ in robots.txt?

WordPress already adds Disallow: /wp-admin/ in its default virtual robots.txt — you don't need to add it again. However, Allow: /wp-admin/admin-ajax.php is also added because some plugins use this URL for legitimate front-end calls. If you customise robots.txt by overriding it, preserve this Allow rule.

How long until Google notices a robots.txt change?

Google fetches robots.txt at the start of every crawl session, typically every 24 hours. Changes propagate within a day. To force faster recognition, use Search Console > robots.txt Tester > Submit. Removing blocks from URLs Google previously couldn't crawl doesn't instantly re-crawl them — that may take days to weeks depending on the URL's discovery priority.

How to Fix Robots.txt Blocks

An accidental Disallow in robots.txt is the fastest way to deindex content. A single line — Disallow: / — can de-rank an entire site. The Site Crawler flags URLs it couldn't access because robots.txt blocked them. This guide walks through finding accidental blocks, fixing them safely, and the staging-environment leak that causes 80% of production robots.txt disasters.

1. Audit your current robots.txt

Step 1

Fetch the file

curl https://yourdomain.com/robots.txt

Read every Disallow line. For each, ask: what URLs does this block, and was that intentional?

Step 2

Check for the catastrophic patterns

Look for these red flags:

# Blocks everything from all crawlers — disaster on production
User-agent: *
Disallow: /

# Blocks Google specifically — usually a development leak
User-agent: Googlebot
Disallow: /

# Blocks everything except homepage — common staging artefact
User-agent: *
Disallow: /
Allow: /$

If any of these are in production, immediately fix.

2. Cross-reference with Site Crawler findings

Step 1

Run the crawler with robots-respect enabled

Run the Site Crawler. It reports any URLs it discovered but couldn't fetch because robots.txt blocked them. Compare these to your intent: which were meant to be blocked (admin, internal endpoints), and which were unintentionally blocked?

3. The staging-config leak

⚠️ Most common production disaster: a development team adds Disallow: / to staging robots.txt to keep crawlers out. A deployment pipeline copies the staging robots.txt to production. Site goes invisible overnight.

Step 1

Verify production robots.txt is the right one

Compare your production robots.txt against your repo. If they don't match, find the deployment step that's copying the wrong file. Common culprits: CI pipeline copying robots.staging.txt regardless of environment, Docker container using a staging entrypoint script in production.

Step 2

Environment-aware robots.txt

Best practice: serve robots.txt dynamically based on environment. Example for Next.js:

// app/robots.ts
export default function robots() {
  const isProd = process.env.VERCEL_ENV === 'production';
  return isProd
    ? { rules: [{ userAgent: '*', allow: '/' }], sitemap: 'https://yourdomain.com/sitemap.xml' }
    : { rules: [{ userAgent: '*', disallow: '/' }] };
}

Production allows; staging/preview blocks. No file-copy gymnastics required.

4. Fix accidental scope errors

Common error 1: trailing wildcards too broad

# Intended to block ?print=true
Disallow: /*?

# Actually blocks every URL containing a query string
# Fix:
Disallow: /*?print=*

Common error 2: blocking CSS/JS

# Old-school pattern that no longer makes sense
Disallow: /wp-includes/

# This blocks Google from rendering your pages — they need CSS/JS to evaluate UX.
# Fix: remove the block, or add explicit allows:
Allow: /wp-includes/*.css
Allow: /wp-includes/*.js

Common error 3: case sensitivity

Robots.txt is case-sensitive. Disallow: /Admin/ doesn't block /admin/. If your CMS routes are case-insensitive, you may think you've blocked something that's actually open.

5. Scope blocks correctly for what really should be blocked

Step 1

A clean production robots.txt template

User-agent: *
Allow: /

# Block administrative paths
Disallow: /admin/
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Block search results and parameter combinations
Disallow: /search?
Disallow: /*?session=

# Block crawl traps
Disallow: /calendar/*/*

Sitemap: https://yourdomain.com/sitemap.xml

6. Test in Search Console

Step 1

Use the robots.txt Tester

Search Console → Settings → robots.txt (the legacy tester is being phased out). Or use the URL Inspection tool: paste an affected URL, look at the "Crawl allowed" line. Should say "Yes". If "No: blocked by robots.txt", review the rule that's blocking it.

Step 2

Request re-crawl

Once your robots.txt is fixed, use Search Console's URL Inspection tool on key pages. Click Request Indexing. Google re-crawls within 24-72 hours typically.

7. Re-run the Site Crawler

Verify the blocked-URL count has dropped. Remaining blocks should all be intentional.

🕷 Re-run the Site Crawler

Verify accidental robots blocks are gone.

Run Site Crawler →