AI Bot Log Analyser: GPTBot and ClaudeBot Activity

Every other AI-visibility signal is inferred. A bot log is observed. When 2.4 million lines of access log say that GPTBot made 847 requests and received 847 responses with status 200, that is not a model, an estimate or a probability — it is a record of what happened, written by your own server at the moment it happened.

This matters because the alternative sources of truth are all partial. Search Console reports on Googlebot and only Googlebot; it has no opinion about Anthropic, OpenAI or Perplexity and never will. Third-party visibility tools query assistants and count mentions, which measures output rather than access, and cannot distinguish "we were never crawled" from "we were crawled and not chosen". A robots.txt checker confirms what you permitted, not what your infrastructure actually did — and, as this report shows, those are frequently different things.

The gap between permission and reality is the entire value of log analysis. Your robots.txt allows ClaudeBot. Your logs contain zero ClaudeBot requests. Both statements are true, and only one of them describes what is happening to your site.

A 403 is a bad answer. Zero log lines is no answer at all, and it is harder to diagnose precisely because there is nothing to find. If ClaudeBot were being refused by the web server, the refusal would be logged: a line with the user-agent, the URL and the status 403. The absence of any line means the request never reached the web server. Something upstream terminated it.

The usual culprits, in the order worth checking:

Verify the fix the same way you found the problem: request a page with the crawler's user-agent from outside your network, confirm a 200, and then confirm the request appears in the log. A rule change that you have not observed in the log has not been proven to work.

The word "AI bot" hides a distinction that determines whether a robots.txt line costs you traffic or costs you nothing. Training crawlers build model corpora. Search agents fetch pages so an assistant can cite them, with a link, to a user who is reading right now. These are different agents with different names, run for different purposes.

Two corrections follow directly from that table. Blocking GPTBot does not remove you from ChatGPT; it removes you from a training set. And disallowing Google-Extended has no effect on Google Search or on AI Overviews, which are served through ordinary Googlebot access — so a site can decline Gemini training without touching its Search visibility at all.

The headline is 1,406 AI bot requests across 28 days. The useful information is where those requests went. GPTBot spent 124 of its 847 fetches on a single blog post and 28 on the pricing page. That distribution is a statement about what your site looks like from the outside: the crawler follows links and sitemaps, and it lands most often where your internal linking sends it.

Three readings are worth doing on any crawl distribution. First, are the heavily crawled pages your best pages? If the corpus being ingested from your site is dominated by thin or stale posts, that is a content problem that no bot-configuration change will fix. Second, are your commercially important pages present at all? A features page fetched 98 times and a pricing page fetched 28 times suggests both are reachable, which is the minimum bar. Third, is anything conspicuously missing — a section of the site the crawler never touches usually turns out to be orphaned, paginated behind a JavaScript control, or excluded by a robots.txt rule nobody remembers writing.

Response time deserves a line of its own. The average here is between 156ms and 212ms, which is comfortable. When a crawler consistently waits several hundred milliseconds or more, it fetches fewer pages in the time it allocates to your host. Slow responses do not get you blocked; they get you sampled thinly, which produces the same practical result — less of your site is read.

A user-agent string is a claim, not an identity. Anybody can send a request announcing itself as GPTBot, and scrapers routinely do exactly that to inherit whatever allowances a site grants to well-behaved crawlers. Before you use log data to make a decision — and certainly before you widen a firewall rule — confirm that the requests came from the operator they claim to come from.

Two mechanisms are available. The major operators publish IP ranges for their crawlers, and matching the source address of a request against the published list is the simplest check. Google additionally supports reverse DNS verification: a reverse lookup on the requesting IP should resolve to a Google-owned hostname, and a forward lookup on that hostname should resolve back to the same IP. A user-agent claiming to be Googlebot from an address that fails that round trip is not Googlebot.

This is also the correct answer to the question that follows every log report: "should I block the traffic I do not recognise?" Not on the basis of the user-agent alone, and not with a rule broad enough to catch legitimate crawlers as collateral. The ClaudeBot silence in this report is exactly what that kind of rule produces.