Fixing Robots Meta Directives

How the robots meta tag works

The tag sits in the head and holds one or more directives separated by commas:

<meta name="robots" content="noindex, nofollow">

Three things decide how it is applied:

• Everything is allowed by default. A page with no robots tag can be indexed, its links followed and its content shown in snippets. That means index, follow does nothing; it is harmless, but you never need to add it.
• The name chooses the crawler. name="robots" applies to all search engines. name="googlebot" applies only to Google's crawler, and name="bingbot" only to Bing's. A crawler obeys the general tag and its own tag together.
• The most restrictive rule wins. If one tag allows indexing and another forbids it, the page is treated as forbidden. Google applies the combination of all the directives it finds, from the meta tags and from the HTTP header.

A crawler can only obey a tag it can read, which means it has to be allowed to fetch the page. If robots.txt blocks the address, the noindex is never seen. That trap has its own guide: noindex and disallow clashes. This page is about the directives themselves.

Every directive, and when to use it

These are the values Google documents for the robots meta tag and the X-Robots-Tag header. Bing supports most of them too. Values are not case sensitive.

noarchive still appears in many templates. Google removed its cached page links in 2024 and says it no longer uses this directive, though other search engines such as Bing still may. Leaving it in place is harmless; adding it for Google has no effect.

What the tag cannot do

The robots meta tag controls search results. It does not stop anyone reading the page, it does not remove a page from AI training datasets, and there is no Google-documented meta value for that. Google's control for its AI models, Google-Extended, is a robots.txt user agent, not a meta directive. Unofficial values such as noai are not part of Google's documented list, so do not rely on them.

Preview directives that quietly hold pages back

An accidental noindex is easy to spot once you look, because the page disappears. Preview limits are harder, because the page still ranks; it simply appears with less.

max-image-preview

Without max-image-preview:large, Google may show only a small thumbnail next to a result, or none at all in some features. Google recommends allowing large previews for pages that want to appear in Discover with a large image. WordPress has added max-image-preview:large to its robots tag by default since version 5.7, but custom themes, headless setups and other platforms usually do not. For most public pages, the recommended tag is:

<meta name="robots" content="max-image-preview:large">

max-snippet and nosnippet

Some templates ship with max-snippet:50 or a similar small value copied from a publisher's setup. That limits every text snippet to a few words. Google has also said its snippet controls apply to how content is shown in its AI features in Search, so a very tight limit may reduce how much of the page can be quoted there. Unless you have a licensing reason, remove small limits or use max-snippet:-1.

To keep one passage out of snippets without limiting the whole page, mark it with data-nosnippet on a span, div or section element instead.

noimageindex

Where unexpected directives come from

Robots rules are rarely typed into pages by hand. When the analyzer flags one nobody expected, it has almost always come from one of these places.

JavaScript deserves care. Google has said that if the served HTML contains noindex, it may skip rendering the page, so a script that later removes the noindex may never run for Google. Put the final value in the server HTML.

The X-Robots-Tag header

Files that are not HTML, such as PDFs, images and spreadsheets, cannot carry a meta tag. The same directives can be sent as an HTTP header instead, which also makes it possible to apply one rule to many files at once.

# Apache (.htaccess or virtual host): keep PDFs out of search
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

# nginx: the same rule
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex";
}

Because the header never appears in the HTML, it is the usual explanation when a page will not index and its head looks clean. Check the headers directly:

curl -sI https://www.example.co.uk/services/ | grep -i x-robots-tag

A header can also name a crawler, for example X-Robots-Tag: googlebot: nosnippet. Keep robots rules in one place where you can. When some are in templates and others in server config, nobody has the full picture.

A sensible default for each page type

The fix, step by step

Stopping stray directives coming back

• Check indexing after every launch and migration. The staging noindex going live is one of the most common causes of a site vanishing from search. Make "robots tag checked on live site" a launch checklist item.
• Monitor a few key pages. A scheduled check that fetches your home page and top pages and alerts if noindex appears in the HTML or headers catches accidents within hours rather than weeks.
• Keep one owner for robots rules. Decide whether the CMS, the SEO plugin or the server sets them, and document any exceptions.
• Review after plugin and theme updates. Content type settings can reset when plugins are reinstalled or replaced. The WordPress meta tags guide covers where each plugin keeps these switches.
• Watch Search Console. A sudden rise in pages excluded by noindex is the clearest early warning you will get.