Excluded by Noindex Tag: How to Find and Remove the Noindex

What the status means

Search Console lists this reason as Excluded by 'noindex' tag. Google crawled the URL, read a noindex directive and left the page out of the index as instructed. Unlike most other reasons in the Page indexing report, this one is entirely under your control: remove the directive and the page becomes eligible again.

A noindex can reach Google in two ways:

URL Inspection tells you which one Google found. Under Page indexing, the line Indexing allowed? reads No, with a note such as 'noindex' detected in 'robots' meta tag or 'noindex' detected in 'X-Robots-Tag' http header. That single line saves a lot of searching.

Many URLs here are meant to be. The noindex beginner tutorial explains which pages should carry it, and the robots meta guide covers every directive and how they combine.

Decide which pages should be indexed

Export the list and sort it. Pages that should normally stay noindexed include:

• Login, account, basket and checkout pages.
• Thank-you and order confirmation pages.
• Internal search results such as /?s= or /search?q=.
• Thin archives you have chosen to keep out, such as tag or date archives.
• Staging, test and preview copies.

Act on the rest: the home page, service and product pages, categories, articles and landing pages. If every page on the site is listed, the cause is almost certainly a single site-wide switch, such as the WordPress visibility setting or a header added for the whole domain. If only one content type is listed, such as all products or all posts, look at the SEO plugin's settings for that type. If a handful of individual pages are listed, check the settings on those pages.

How to find where the noindex comes from

Check the HTML

Fetch the page and look for any robots meta tag:

curl -s https://example.com/services/ | grep -io '<meta[^>]*\(robots\|googlebot\)[^>]*>'

In a browser, use View Page Source rather than the Elements panel and search for noindex. View Source shows what the server sent. The Elements panel shows the page after scripts ran, so compare the two: a noindex added or removed by JavaScript will only show in one of them.

Check the headers

curl -sI https://example.com/services/ | grep -i x-robots-tag
# x-robots-tag: noindex

Headers are easy to miss because nothing on the page shows them. In Chrome DevTools, open the Network panel, reload, click the document request and read the Response Headers. Check redirects too: run curl -sIL to see the headers on every hop, because a header on the final URL is what counts.

Search the server and CDN

If the header is there, find the rule that adds it. On a Linux server:

sudo grep -rin "x-robots-tag" /etc/nginx /etc/apache2 /etc/httpd 2>/dev/null
sudo grep -rin "x-robots-tag" /var/www/vhosts/system/*/conf 2>/dev/null
grep -rin "x-robots-tag" /var/www/vhosts/example.com/httpdocs/.htaccess

The second line covers Plesk, which stores the generated configuration for each domain under /var/www/vhosts/system/. Rules you typed into Plesk's Apache and nginx Settings for a domain end up there. In Cloudflare, look under Rules for Transform Rules that modify response headers. Our robots and sitemap checker reports both the meta tag and the header for a URL in one pass.

Fixing it in WordPress

The site visibility setting

Go to Settings, then Reading, and find Search engine visibility. If the box Discourage search engines from indexing this site is ticked, WordPress adds a robots meta tag with noindex, nofollow to every page. It is often left ticked when a site moves from staging to live. Untick it and save.

The SEO plugin

Plugins can noindex at three levels, and you need to check all of them:

• Per page. In Yoast SEO, open the post, go to the Advanced tab in the Yoast panel and check the question Allow search engines to show this Post in search results?, where the word Post changes with the content type In Rank Math, open the Advanced tab and check that No Index is not selected under Robots Meta.
• Per content type. In Yoast, the Settings area has a section for each content type, such as posts, pages and products, with a Show in search results switch. Rank Math has the same under Titles and Meta for each post type and taxonomy.
• Archives. Categories, tags, author and date archives each have their own switch. Turning off author archives on a single-author blog is fine; turning off product categories on a shop is not.

Caches, themes and other plugins

After changing settings, purge every cache: the page cache plugin, the server cache and the CDN. A cached page can keep serving the old noindex for days. If the tag still appears, switch to a default theme briefly on a staging copy, or search the theme for wp_robots filters and hard-coded meta tags. Some membership, maintenance-mode and coming-soon plugins add noindex too. For the wider setup, see the WordPress robots and sitemap guide.

Servers, CDNs and other platforms

nginx

Look for an add_header line. It may apply to a whole server block or only to a location:

location /blog/ {
    add_header X-Robots-Tag "noindex";   # remove this line
}

Remove or narrow it, then test and reload with sudo nginx -t && sudo systemctl reload nginx.

Apache and .htaccess

<IfModule mod_headers.c>
  Header set X-Robots-Tag "noindex, nofollow"
</IfModule>

Remove the Header set line, or limit it to file types you really want kept out, such as PDFs, using a FilesMatch block. Changes to .htaccess take effect at once; changes to the virtual host need a reload.

Cloudflare, Vercel and other hosts

In Cloudflare, check Transform Rules for response header changes. Preview and staging addresses on some hosting platforms, such as Vercel preview deployments and Cloudflare Pages preview URLs, send X-Robots-Tag: noindex by design. That is correct for previews, but make sure your live site is served from the production domain, not a preview address.

Shopify and Next.js

Shopify has no noindex switch in the admin, so a noindex usually comes from theme code in theme.liquid or an app. Search the theme for noindex. In Next.js, check the robots field in the metadata of the layout and each route; a robots: { index: false } set in the root layout applies to every page beneath it.

A worked example

A plumbing firm launches a new WordPress site. Three weeks later the Page indexing report shows 46 pages excluded by noindex tag, which is every page on the site. URL Inspection on the home page says 'noindex' detected in 'robots' meta tag, and View Source shows:

<meta name='robots' content='noindex, nofollow' />

Under Settings, then Reading, the box Discourage search engines from indexing this site is ticked, left over from the staging build. The owner unticks it, purges the page cache and the Cloudflare cache, and checks again with curl: the tag now reads index, follow from the SEO plugin. A live test in URL Inspection shows Indexing allowed? Yes. Indexing is requested for the home page and the six main service pages, and Validate fix is started on the issue. Within two weeks most pages are indexed.

The fix, step by step

Edge cases worth knowing

• JavaScript cannot rescue a noindex. If the server HTML contains noindex, Google may skip rendering the page, so a script that removes the tag later is not reliable. Fix it in the server HTML.
• Conflicting tags. If a page carries both index and noindex, Google applies the more restrictive one.
• Bot-specific tags. A googlebot meta tag overrides nothing for other search engines, and a bingbot one has no effect on Google.
• Noindex in the sitemap. Listing noindexed URLs in your sitemap sends mixed signals. Remove them, or remove the noindex.
• Non-HTML files. PDFs and images can only be noindexed with the header, since they have no HTML head.

Keeping it fixed

Most accidental noindexes arrive during a launch, migration or plugin change. Add a check to every launch list: open the home page and one page of each type, and confirm both the meta tag and the header allow indexing. Run a site crawl monthly and list any indexable page type that carries noindex. Keep staging sites behind a password rather than relying only on noindex, so the setting is never copied to the live site. If you move hosts or domains, the WordPress migration guide lists the settings that most often slip through.