Missing From ChatGPT Answers? Diagnose Crawler, Index, Brand
Your competitors turn up when people ask ChatGPT for a recommendation, and your business does not, even when you ask about it by name. That absence nearly always has a findable cause.
Either OpenAI's search crawler is being turned away somewhere between the internet and your pages, or the pages it would need are not reachable through search, or ChatGPT cannot tell your business apart from something else. This guide is a diagnosis: it shows how to test which of those applies, how to prove it from your logs, and how to remove each blocker.
What "invisible in ChatGPT" can mean
Before changing anything, pin down what kind of absence you are dealing with. "ChatGPT never mentions us" covers several different situations, and each points to a different layer of the problem.
ChatGPT can reply in two ways. Sometimes it answers purely from what the model learned in training, with no links at all. At other times it runs a web search, reads some pages and shows its sources. You can usually tell which happened: a searched answer shows source links or a sources panel, while a trained-knowledge answer does not. Only the searched kind can be influenced quickly, because the model's trained knowledge is fixed until OpenAI releases a newer model.
| What you see | Most likely layer | Where to look first |
|---|---|---|
| ChatGPT searches, but never links to your domain, even for your own brand name | Access: OpenAI's search crawler cannot fetch the site | robots.txt, CDN bot settings, access log |
| ChatGPT links to directories or review sites about you, but never your own site | Access or indexing of your own pages | Access log, then whether the pages are indexed |
| ChatGPT describes a different business with the same or a similar name | Identity: your brand is ambiguous | Name, address and profile consistency |
| You appear for your brand name but never for "best [service] in [town]" questions | Not an invisibility problem: you are visible but not chosen | The optimisation guide rather than this one |
| Answers without search say nothing, or something out of date | Training data, which you cannot edit directly | Make sure searched answers work instead |
If the fourth row describes you, the site is reachable and known; the job is to become the better source, which is covered in Optimising for ChatGPT. The rest of this guide assumes ChatGPT cannot, or will not, surface you at all.
What OpenAI has published about its crawlers
OpenAI documents its crawlers on its developer site, and that page is the only reliable source for their names and purpose. At the time of writing it lists three user agents, each controlled separately in robots.txt:
OAI-SearchBotis described as the crawler used to surface and link websites in ChatGPT's search features. OpenAI states that sites which opt out of it will not be shown in ChatGPT search answers, though they may still appear as plain navigational links. This is the one that matters most for invisibility.ChatGPT-Useris used when a person's request in ChatGPT, or in a custom GPT, causes a page to be visited. OpenAI describes it as user-initiated rather than an automatic crawler, so read its current documentation for how robots.txt rules apply to it. Treat its presence in your log as a sign of real user demand, not of indexing.GPTBotcollects content that may be used to train OpenAI's models. OpenAI presents it as independent of search, so blocking it is a training decision and should not, by OpenAI's account, remove you from search answers.
OpenAI also says that changes to robots.txt can take around a day to be picked up by its search systems, and it publishes the IP address ranges its crawlers use as downloadable JSON files, which lets you check that a request claiming to be OAI-SearchBot really came from OpenAI.
What OpenAI has not published in detail is exactly how ChatGPT decides which pages to cite. At launch it said ChatGPT search draws on third-party search providers as well as content from its partners and its own crawling, without giving a full list or a ranking formula. Anyone claiming to know the precise weighting is guessing. The practical consequence is simple: make sure OpenAI's own crawler can reach you, and make sure you are well indexed in the major search engines too.
Proving where the block is
Guessing at the cause wastes days, because a change to robots.txt takes time to show in answers. A short, methodical test tells you which layer to fix.
Test the answers, not your impression of them
Use a fresh chat, ideally in a private browser window or with memory switched off, because ChatGPT can personalise replies using past conversations. Ask three kinds of question and make sure search is used for each; you can ask it directly to search the web.
- Navigational: "What is example.co.uk?" or "Tell me about Fernbrook Joinery in Leeds." If a searched answer cannot find or link your own site for your own name, the problem is almost certainly access or indexing.
- Factual about you: "What are Fernbrook Joinery's opening hours?" A wrong or missing answer with other sites cited shows ChatGPT is relying on third-party pages rather than yours.
- Category: "Who makes bespoke staircases in Leeds?" Absence here, with the first two working, is a competitiveness problem, not invisibility.
Read the access log
Your server log is the ground truth. Download the last seven to thirty days and count requests from each OpenAI user agent together with the status codes they received.
grep -E "OAI-SearchBot|ChatGPT-User|GPTBot" access.log \
| awk '{print $9}' | sort | uniq -c | sort -rn
# Which pages did OAI-SearchBot fetch, and what did it get?
grep "OAI-SearchBot" access.log | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -30The field numbers assume the common combined log format; adjust them if your host uses another layout. Read the result like this:
| Log result | Meaning |
|---|---|
| No OAI-SearchBot lines at all | It is being stopped before it reaches your server (CDN or host firewall), or robots.txt forbids it and it is obeying, or it has not yet discovered the site. |
Only requests for /robots.txt | It is reading your rules and being told to stay away. Read the file. |
| Status 403, 406 or 503 on pages | Your server, a security plugin or a bot rule is refusing it. |
| Status 429 | Rate limiting is treating it as abusive traffic. |
| Status 200 on the pages you care about | Access is fine. Move on to indexing, content and identity. |
If your host only gives you the log after a CDN, check the CDN's own security or firewall event log too. A request blocked at the edge never reaches the origin server, so it will not appear in the log you downloaded.
Check robots.txt the way a crawler reads it
Crawlers follow the most specific group that names them, and if no group names them they follow the
User-agent: * group. A file with a named group for OAI-SearchBot that says Disallow: /
blocks it even if the wildcard group allows everything. Equally, a wildcard Disallow: /
left over from a staging site blocks every crawler that has no group of its own.
curl -s https://www.example.co.uk/robots.txtRead the live file on the live domain, including the www and non-www versions, because they can differ.
The fix, step by step
Step 1
Record a baseline across three kinds of prompt
Run the AI Visibility Tracker with a short list of questions your customers ask, then repeat the same set by hand in a fresh ChatGPT chat with search in use: one navigational question naming your business, one factual question about your opening hours, prices or services, and one category question about your service and town. Note for each whether you are cited, only mentioned, or missing, and save the date so later changes can be compared fairly.
Step 2
Confirm what OpenAI crawlers actually receive
Download at least a week of server access logs and search them for OAI-SearchBot,
ChatGPT-User and GPTBot. Count the status codes each one received and list
the pages they requested. Then check the security event log of your CDN or host for the same names. Zero
requests, requests only for robots.txt, or 403, 429 and 503 responses each point to a specific block
that you can now name and remove with confidence.
Step 3
Remove the block at every layer
Fix
the layer the logs identified, then check the others anyway, because blocks are often stacked. In robots.txt,
give OAI-SearchBot its own group with Allow: / and remove any wildcard Disallow: /
left from staging. In the CDN, turn off blanket AI-crawler blocking or add OpenAI's search crawler to
the allow list. In security plugins and rate limits, stop challenging it. Decide about GPTBot
separately.
Step 4
Check the pages are indexable and readable as HTML
For each page you want cited, fetch it with curl -A "OAI-SearchBot"
and confirm it returns status 200 and that the answer text appears in the raw HTML rather than only after
scripts run. Look for a stray noindex in the meta robots tag or the X-Robots-Tag header,
a canonical pointing to another address, and cookie or consent screens that replace the content. Remember
a user agent test cannot reproduce IP-based blocking.
Step 5
Make sure search engines have found the site
OpenAI says ChatGPT search also draws on third-party search providers, so a site the major engines do not know is at a disadvantage. Verify the site in Google Search Console and Bing Webmaster Tools, submit an up-to-date XML sitemap in both, and inspect the key URLs to confirm they are indexed. If they are not, fix that first, because no amount of work on answers will help pages that search engines have never seen or have excluded.
Step 6
Give ChatGPT one unambiguous version of your business
If searched answers describe a different company, make your identity impossible
to confuse. Use exactly the same business name, address, phone number and web address on your site, your
Google Business Profile and the main directories. Add Organization markup with sameAs links
to your real profiles, and write a plain About page that says who you are, where you trade and what you
do. Where a name clash exists, add your town or trade to how you describe yourself.
Step 7
Retest once the changes have had time to land
Wait at least a few days after the fix, since OpenAI says robots.txt changes take around a day to register and discovery takes longer, then confirm in the log that OAI-SearchBot now receives status 200 on your key pages. Repeat the exact baseline prompts from step 1 in a fresh chat and run the tracker again. Compare like with like, and judge the trend over several weeks rather than a single reply.
Cases this does not fix, and what to do instead
Some absences are not caused by anything on your site, and it helps to recognise them before spending time on the wrong layer.
- ChatGPT answered without searching. If the reply has no sources, it came from the model's training. Nothing you publish today changes that answer until a future model is trained. What you can do is make sure that when ChatGPT does search, it finds you; asking for current information, prices or local options tends to make search more likely.
- ChatGPT is confident but wrong about you. That is a correctness problem rather than invisibility. The wrong attribution guide covers tracing the source it copied and correcting it.
- Brand-new site or domain. A site that went live recently may simply not be discovered yet. Getting linked from a few places that are already crawled, and being indexed in the main search engines, helps more than waiting.
- You deliberately block AI. If you have chosen to block all AI crawlers, invisibility is the expected result. You can still allow OAI-SearchBot for search while blocking GPTBot for training, if that split suits you.
- Pages behind a login or paywall. Crawlers see what an anonymous visitor sees. Content only members can read will not be cited, though a public summary page can be.
- Personalised replies. Answers inside a logged-in account can be shaped by that person's memory and past chats, so a colleague may see you where you do not, or the reverse. Always test in a clean session.
Checking for ChatGPT visits in analytics
Your analytics can confirm that the fix worked from the visitor side. OpenAI has said that links in
ChatGPT search results carry a utm_source=chatgpt.com parameter so that publishers can see
the traffic, and some visits also arrive with chatgpt.com as the referrer. In Google Analytics 4, look
under Reports, Acquisition, Traffic acquisition, and add session source as a dimension; filter for chatgpt.
Two cautions. First, not every visit that starts in ChatGPT is labelled, because some people copy the address or open it in another app, so treat the figure as a floor, not a total. Second, zero visits does not prove invisibility, especially for a small business where a handful of people may ask about you each month. Use the log and the prompt tests as the main evidence and analytics as a supporting signal.
# Access log: requests that arrived from ChatGPT links
grep -E "utm_source=chatgpt.com|chatgpt.com" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | headStopping it happening again
Invisibility caused by a block tends to come back, because the thing that caused it, a plugin update, a CDN default, a redesign, is still there. A few habits catch it early:
- Keep robots.txt under version control or at least keep a dated copy, and check it after every theme change, migration or plugin update.
- Set a simple log alert. A weekly count of OAI-SearchBot requests and their status codes, emailed or viewed in a dashboard, shows a new block within days rather than months.
- Read CDN change notes. Bot-management features are evolving quickly, and a provider can add new AI-crawler defaults. When one is announced, check what it does to your settings.
- Document your AI crawler policy. Write down which crawlers you allow and why, so that the next developer or agency does not "tidy up" robots.txt and block search along with training.
- Re-run the same prompts monthly. Use the AI Visibility Tracker with a fixed set of questions so a sudden drop stands out from normal day-to-day variation.
For the underlying robots.txt and firewall rules across every assistant, not just ChatGPT, see fixing AI crawler access.
1. Why does ChatGPT not show my website when I ask about my own business?
When a searched answer cannot find your own site even for your own name, the usual cause is access. OAI-SearchBot, the crawler OpenAI says it uses for ChatGPT search, may be disallowed in robots.txt, refused by a firewall or security plugin, or blocked by a CDN setting aimed at AI crawlers. Check your access log for OAI-SearchBot and the status codes it received. If access is fine, confirm the pages are indexed in Google and Bing and that your business name is not easily confused with another.
2. Does blocking GPTBot remove me from ChatGPT?
According to OpenAI's documentation, no. GPTBot is described as the crawler that gathers content which may be used for training models, while OAI-SearchBot is the one used to surface and link websites in ChatGPT search. Each is controlled by its own group in robots.txt. You can block GPTBot and still allow OAI-SearchBot. Check the current OpenAI bots page before making the change, because the names and descriptions are set by OpenAI and can be updated over time.
3. How long after unblocking OAI-SearchBot will I appear in ChatGPT?
There is no published timetable for appearing in answers. OpenAI says robots.txt changes take roughly a day to be picked up by its systems, but the crawler then has to visit your pages, and ChatGPT has to judge them useful for a given question. Confirm in the access log that OAI-SearchBot starts receiving status 200 on your key pages within the first week. After that, retest with the same prompts over several weeks rather than expecting an immediate change.
4. ChatGPT mentions my business but never links to my website. Why?
That usually means ChatGPT knows about you from other sources, such as directories, review sites or news articles, but is either unable to fetch your own pages or finds them less useful for the question asked. First rule out access by checking the log for OAI-SearchBot. If it is fetching your pages successfully, compare what your page says with the pages ChatGPT did link. Often the third-party page answers the specific question more directly than your own site does.
5. Can I check whether a request really came from OpenAI's crawler?
Yes. Anyone can put OAI-SearchBot in a user agent string, so the name alone proves nothing. OpenAI publishes the IP address ranges its crawlers use as JSON files linked from its bots documentation. Compare the IP address in your log line with those ranges. Several CDNs also maintain lists of verified bots and label genuine crawler traffic in their dashboards. Only allow-list by IP range if you keep the list updated, because providers change their ranges.
6. Is it worth submitting my site to Bing if I want to be in ChatGPT?
It is a sensible precaution rather than a guaranteed route. OpenAI has said ChatGPT search uses third-party search providers alongside its own crawling, without publishing a full list. Being verified and indexed in Bing Webmaster Tools and Google Search Console costs little, lets you spot indexing problems early, and helps with other assistants too. It will not make up for OAI-SearchBot being blocked, so fix direct access first and treat search engine indexing as the second check.
7. Why does a colleague see my business in ChatGPT when I do not?
ChatGPT answers are generated fresh every time, so the same question can produce different sources on different runs. Logged-in answers can also be shaped by a person's saved memory and past conversations, and one person may have phrased the question in a way that triggered a search while the other did not. Test in a clean, private session, repeat each prompt several times, and use a tracker that samples the same questions regularly rather than relying on individual screenshots.
The AI Visibility Tracker asks AI assistants the questions your customers ask and records whether your brand is cited, mentioned, recommended or missing, so you can confirm the problem before you fix it and track the change afterwards.
Open AEO Fixes →