What is GPTBot, and does ChatGPT crawl websites?

15 October 2026 · By John Carlsson

GPTBot is OpenAI's crawler for content that may be used to train its generative AI models. Yes, ChatGPT reaches websites, but through other agents: OAI-SearchBot crawls for ChatGPT search, and ChatGPT-User visits a page when a user's question needs it. Each is controlled separately in robots.txt, so blocking GPTBot does not remove you from ChatGPT search.

OpenAI's three main user agents

OpenAI documents its crawlers on one overview page. The three you will see most often do different jobs:

GPTBot: used to crawl content that may be used in training OpenAI's generative AI foundation models. Its user agent string includes GPTBot/1.4; +https://openai.com/gptbot.
OAI-SearchBot: used for search. It surfaces websites in ChatGPT search, and sites that opt out will not be shown in ChatGPT search answers. Its string includes OAI-SearchBot/1.4; +https://openai.com/searchbot.
ChatGPT-User: used when a person asks ChatGPT or a custom GPT a question and it visits a web page to answer. Its string includes ChatGPT-User/1.0; +https://openai.com/bot.

OpenAI notes that version numbers in these strings may change, so match on the bot name rather than the full string. The same page also lists OAI-AdsBot, which only visits pages submitted as adverts and is not used for training.

So does ChatGPT crawl websites?

Yes, in two ways. OAI-SearchBot crawls the web automatically to build ChatGPT search. ChatGPT-User fetches individual pages on demand when a user's request needs them. OpenAI says ChatGPT-User is not used for automatic crawling, and that because its visits are started by a user, robots.txt rules may not apply to it.

GPTBot is separate from both. Allowing or blocking it is about training, not about whether ChatGPT can show your page in an answer.

robots.txt examples

Each OpenAI setting is independent. robots.txt lives at the root of your site, and under the Robots Exclusion Protocol each group starts with a User-agent line followed by Allow or Disallow rules.

Stay in ChatGPT search but opt out of training

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

Block everything from OpenAI's automatic crawlers

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /

The second option means your site will not be shown in ChatGPT search answers. Put a blank line between groups in the real file.

Block one folder from training only

User-agent: GPTBot
Disallow: /members/

OpenAI says it can take around 24 hours after a robots.txt update for ChatGPT search results to reflect the change.

Things robots.txt will not do

It is not security. RFC 9309 says the rules are not a form of access authorisation. Anything private needs a login or server-side block.
It does not reliably stop ChatGPT-User, for the reason above.
It does not remove a page from ChatGPT search results on its own if you still want the crawler to see the page. OpenAI's publishers FAQ says to use a noindex meta tag for pages you do not want shown even as a link, and the crawler must be allowed to read that tag.

Checking which bots visit you

Look in your server access logs for the bot names above. To confirm a visit is genuine, compare the IP address with the lists OpenAI publishes for each bot:

openai.com/gptbot.json
openai.com/searchbot.json
openai.com/chatgpt-user.json

If you run a firewall or CDN with bot blocking and you want to appear in ChatGPT search, allow the OAI-SearchBot range. OpenAI's ChatGPT search help says sites must let traffic from those published addresses through to be eligible.

Sources

Frequently asked questions

Does ChatGPT-User follow robots.txt?

Not reliably. OpenAI says that because ChatGPT-User actions are started by a user, robots.txt rules may not apply. It also says ChatGPT-User is not used to crawl the web automatically and is not used to decide whether content can appear in ChatGPT search. To manage search opt outs, OpenAI says to use the OAI-SearchBot user agent in robots.txt.

How can I tell if a visit really came from GPTBot?

Check the IP address, not just the user agent string, because anyone can copy a user agent. OpenAI publishes IP ranges for each bot: openai.com/gptbot.json for GPTBot, openai.com/searchbot.json for OAI-SearchBot and openai.com/chatgpt-user.json for ChatGPT-User. A request claiming to be GPTBot from an address outside the list is not from OpenAI.