Home / Blog / Indexing and Crawling / Is robots.txt still used, and what is the difference between robots.txt and noindex?

Answers

Is robots.txt still used, and what is the difference between robots.txt and noindex?

16 October 2026 · By John Carlsson

Yes, robots.txt is still used, and since September 2022 it has been an IETF standard, RFC 9309. It controls which URLs crawlers may fetch, mainly to manage crawler traffic. noindex controls whether a page appears in search results. A page blocked in robots.txt can still be listed, so to keep a page out of Google, use noindex and leave it crawlable.

Is robots.txt still used?

Robots.txt has been around for a long time, but it became a formal internet standard only recently. In September 2022 the IETF published RFC 9309, the Robots Exclusion Protocol, as a Proposed Standard. It says the rules must sit in a file named /robots.txt, in lowercase, at the top level of the site, encoded as UTF-8 plain text.

Search engines still read it, and so do AI companies. OpenAI documents separate crawlers with separate purposes. Disallowing GPTBot indicates that your content should not be used to train its generative AI foundation models. OAI-SearchBot is used to show sites in ChatGPT search, and disallowing it means your pages will not appear in ChatGPT search answers, although they can still show as navigational links. That lets a site opt out of training while staying visible in search.

RFC 9309 is clear that these rules are not a form of access authorisation. Reputable crawlers follow them, but nothing forces a crawler to do so.

What is the purpose of a robots.txt file?

Google says robots.txt tells crawlers which URLs they can access on your site, and that it is used mainly to manage crawler traffic so your server is not overloaded. It can also keep crawlers away from unimportant or near duplicate pages.

What it is not, in Google's words, is a mechanism for keeping a web page out of Google. A blocked URL can still be indexed if other pages link to it, and it can appear in results without a description. It is not a security measure either, so private content needs password protection on the server.

What is the difference between robots.txt and noindex?

The short version is that robots.txt controls crawling, and noindex controls indexing.

robots.txt asks crawlers not to fetch certain URLs. It does not remove those URLs from results.
noindex tells Google not to show a page in results. Once Googlebot sees the rule, it drops the page, even if other sites link to it.

You can set noindex in a meta tag in the page head, or as an HTTP response header, which also works for files such as PDFs:

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The two must not be combined on the same URL. Google warns that if a page is blocked in robots.txt, the crawler never sees the noindex rule, so the page can still appear in results when other pages link to it. Google also does not support noindex rules inside robots.txt itself.

How do you prevent Google from crawling your site?

Google points to robots.txt as the way to stop crawling. A rule that blocks Googlebot from the whole site looks like this:

User-agent: Googlebot
Disallow: /

Think about the goal before adding it. Google points out that blocking crawling does not stop a URL appearing in results, and that it is almost impossible to keep a site secret just by not linking to it, because URLs leak through links and referrer logs. Blocking Googlebot also affects Discover and other Search features, plus products such as Google Images and Google News.

So choose the method by outcome:

To reduce crawling, use robots.txt.
To keep a page out of results, use noindex and leave the page crawlable.
To keep content private from everyone, use password protection or remove it.

After adding noindex, use the URL Inspection tool in Search Console to see what Googlebot received, and check the Page Indexing report for pages where a noindex rule was found.

Sources

Frequently asked questions

Can I put a noindex rule in robots.txt?

No. Google states that it does not support noindex rules in the robots.txt file. To keep a page out of results, add a robots meta tag with noindex to the page head, or return an X-Robots-Tag: noindex HTTP header. The page must stay crawlable, because Googlebot has to fetch it to see the rule. Blocking it in robots.txt at the same time stops the rule being read.

How can I check whether Google has seen my noindex rule?

Google has to recrawl a page before it sees a new noindex tag or header, which can take a while for less important pages. Use the URL Inspection tool in Search Console to view the HTML Googlebot received and to request a recrawl. The Page Indexing report also lists pages where Googlebot found a noindex rule. Check robots.txt as well, since a block there hides the rule.