Fixing AI Agents Blocked by Firewalls and Bot Protection
Plenty of sites turn away AI agents without anyone choosing to. A firewall, a CDN security setting or a WordPress security plugin sees an automated request from a cloud data centre, decides it is a scraper and answers with a challenge page, a 403 or an endless redirect.
The agent working for a real customer then reports that your site is unavailable and suggests a competitor. This guide shows how to prove which layer is doing the blocking, how to recognise agents that identify themselves, and how to write narrow allow rules that let them in while your protection against abuse stays switched on.
- What a blocked agent looks like from both ends
- The layers that can refuse an agent
- How agents identify themselves
- Cloudflare, AWS WAF and WordPress specifics
- The fix, step by step
- Commands for testing from the outside
- Keeping protection without collateral damage
- Stopping it from happening again
- Questions and answers
What a blocked agent looks like from both ends
From the customer's side, the symptom is vague. They ask an assistant to check whether a product is in stock or to book a slot, and it replies that the site "could not be accessed", "returned an error" or "appears to be down". Some agents quietly move on to another site without saying why. You almost never hear about it, because the person never reached you.
From your side, the evidence sits in logs you may not normally read:
- A 403 Forbidden from the firewall or the web server, often with a branded error page from the CDN.
- A 503 or a JavaScript challenge page, such as an interstitial that says the browser is being checked. A fetch-based agent cannot run that script, and a browser-based agent may be looped back to it repeatedly.
- A 429 Too Many Requests when a rate limit has been set so low that an agent loading a page plus its scripts and images trips it.
- A CAPTCHA on the very first page. Reputable agents will not solve these and will hand the task back to the person or give up.
- A 200 response that is not your page, for example a consent wall, a country selector or a "please enable cookies" message served in place of the content.
The last one is the easiest to miss, because nothing in your monitoring shows an error. Always look at what the body of the response contains, not only the status code.
The layers that can refuse an agent
A request passes through several systems before it reaches your page, and any of them can block it. Working out which one is responsible saves hours of changing the wrong setting.
| Layer | Typical examples | Where to look |
|---|---|---|
| DNS-level CDN and bot management | Cloudflare Bot Fight Mode, Super Bot Fight Mode, managed AI bot blocking; Akamai Bot Manager; Fastly bot rules | The CDN dashboard security events, filtered by user agent or IP address |
| Cloud firewall | AWS WAF with Bot Control, Azure Front Door WAF, Google Cloud Armor | The WAF sampled requests or logs, which show the rule that matched |
| Hosting control panel | Plesk or cPanel ModSecurity rules, Imunify360, fail2ban jails, host-wide anti-bot settings | The panel's firewall or ModSecurity log, and fail2ban-client status |
| Web server | Nginx or Apache rules that deny by user agent, country or IP range | Server configuration files and the error log |
| Application | Wordfence, Sucuri, iThemes or Solid Security, a Shopify bot app, custom middleware | The plugin's live traffic or blocked requests view |
| Content gate | Age gates, cookie walls, geo redirects, "select your country" pages | Fetch the page with no cookies and read what comes back |
A simple rule of thumb: if your origin server's access log shows no request at all when the agent visits, the block is happening before the server, at the CDN or cloud firewall. If the request is logged with a 403 or 429, it is the server or the application.
How agents identify themselves
Not all AI traffic is the same, and treating it as one thing is the root cause of most accidental blocks. Providers generally run separate systems for separate jobs:
- Training crawlers collect pages in bulk to train models. Examples are
GPTBot,ClaudeBotandCCBot. Blocking these is a content decision and does not stop a customer's agent. - Search and index crawlers build the index an assistant searches, such as
OAI-SearchBot,Claude-SearchBotandPerplexityBot. - User-triggered fetchers visit because one person just asked about your page, such
as
ChatGPT-User,Claude-UserandPerplexity-User. These are the ones that matter for this guide. - Browser-driving agents run a full browser, either in the provider's cloud or on the person's own computer. Cloud ones often send a normal-looking Chrome user agent, which is why a user agent rule alone will not always catch them.
Names and details change, so check each provider's current documentation rather than trusting a list copied from a blog post, including this one.
Three levels of proof
- User agent string. Easy to read, trivially faked. On its own it should never be enough to skip security.
- User agent plus published IP ranges. Several providers publish JSON files listing
the address ranges their fetchers use. A request that claims to be
ChatGPT-Userand comes from OpenAI's published range is far more believable. Reverse DNS lookups serve the same purpose for some crawlers. - Signed requests. Web Bot Auth, being worked on at the IETF, has the agent sign each
request using HTTP Message Signatures. The request carries
Signature,Signature-Inputand aSignature-Agentheader pointing to where the agent's public keys are published. Your CDN checks the signature against those keys. Cloudflare can verify these for you, and some providers sign their agent traffic already. This is the strongest check available and the only one that works well for cloud browsers with ordinary user agents.
Cloudflare, AWS WAF and WordPress specifics
Cloudflare
Cloudflare is where most accidental agent blocks start, because several features overlap. Check each of these under the security settings for the zone:
- Bot Fight Mode on the free plan challenges traffic it thinks is automated. Cloudflare documents that it cannot be bypassed with a WAF skip rule, so if it is catching agents your options are to switch it off or upgrade to a plan with Super Bot Fight Mode, which can be tuned.
- Super Bot Fight Mode has separate choices for definitely automated traffic and for verified bots. Leaving verified bots allowed is usually right.
- AI bot blocking settings let you block AI crawlers across the zone. Check whether yours covers only training crawlers or also user-triggered fetchers, and adjust so it matches what you actually want.
- Custom rules and managed challenges added in a hurry during an attack, such as challenging every visitor from outside the UK, are often forgotten and left running for months.
Cloudflare exposes fields such as cf.client.bot and a verified bot category that can
be used in custom rules. A skip rule placed above your challenge rules can let verified agent traffic
past, while everything else still meets the challenge.
AWS WAF
The Bot Control managed rule group labels requests by bot category and whether the bot was verified. Rather than removing the whole rule group, override the specific rule that is matching to count mode, then add your own rule that blocks based on labels for unverified bots only. The sampled requests view shows exactly which rule and label matched each request.
WordPress security plugins
Wordfence, Solid Security and similar plugins have options such as "block fake Google crawlers", rate limits for crawlers, and country blocking. Look at the live traffic view while the audit runs. Raise the crawler rate limit to something reasonable and make sure the action for exceeding it is throttle rather than block, so a single burst does not ban an address for hours.
The fix, step by step
Step 1
Confirm the block and capture the response
Run the AI Agent Compatibility audit on the page an agent would need, such as a product
or booking page, and note whether it reports a block. Then repeat the fetch yourself from a server or
cloud machine rather than your office connection, sending a user agent such as ChatGPT-User
with curl -sS -A. Save the status code, the response headers and the first part of the body,
because the headers usually name the system that answered, such as a CDN ray ID or a WAF request ID.
Step 2
Find which layer is refusing the request
Search the origin access log for the time of your test. No log line means the CDN or cloud firewall stopped it before your server. A logged 403 or 429 points to the server configuration, a ModSecurity rule or a plugin. Open the security event view of each layer in turn and filter by your test IP address until you find the rule that matched. Write down the rule name or ID so the change you make is targeted, not a guess.
Step 3
Separate user-triggered agents from training crawlers
Decide your policy for each kind of AI traffic instead of treating it as
one group. Most businesses that sell or take bookings want user-triggered fetchers such as ChatGPT-User,
Claude-User and Perplexity-User to succeed, whatever they decide about training
crawlers like GPTBot and ClaudeBot. Check each provider's current documentation
for its crawler names and published IP range files, and record the decision so the next person who changes
a security setting knows it was deliberate.
Step 4
Write a narrow allow rule with verification
Create a skip or allow rule that matches only verified agent traffic: the CDN's verified
bot flag, a valid Web Bot Auth signature, or a user agent combined with the provider's published IP ranges.
Place it above the rule that was blocking, and make it skip only the bot challenge, not every protection.
Never allow on user agent alone, because anyone can send ChatGPT-User from a laptop. Keep
the rule in version control or document it with a date.
Step 5
Replace blanket challenges with rate limits
Look for rules that challenge whole countries, whole cloud providers or every request
without cookies, and replace them with limits on behaviour. A sensible limit counts requests per IP address
per minute on expensive paths such as search, login and basket, and slows down or challenges only the
addresses that exceed it. Return 429 with a Retry-After header rather than
403, so well-behaved agents know to wait and try again rather than report the site as broken.
Step 6
Retest and watch the security log
Repeat your curl test and run the audit again, then ask an agent you have access to for a real task on the site, such as checking a price. Confirm the request reaches the origin with a 200 and your real page in the body. Keep the security event view open for a week and check the new rule's match count, so you can see both genuine agent traffic passing and any sign that someone is abusing the allowance.
Commands for testing from the outside
Test from a machine outside your own network. Many firewalls trust your office IP address or your logged-in session, so a test from your desk can pass while the rest of the world is blocked. A small cloud server, or a colleague's home connection, gives a truer picture.
# Status code, response size and the headers that name the responder
curl -sS -o /tmp/agent-body.html -D - \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
https://www.example.co.uk/products/oak-side-table
# Does the body contain your real content or a challenge?
grep -i -c "oak side table" /tmp/agent-body.html
grep -i -E "captcha|challenge|checking your browser|access denied" /tmp/agent-body.html
# The same request with a plain browser user agent, for comparison
curl -sS -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36" \
https://www.example.co.uk/products/oak-side-tableIf the browser user agent gets a 200 and the agent user agent gets a 403, a rule is matching on the name. If both fail from a cloud server but work from home, the rule is matching on IP reputation or hosting provider. The user agent strings above are examples of the format; copy the exact current strings from each provider's documentation when you test.
On the origin, this finds any agent requests and the status your server gave them:
grep -E "ChatGPT-User|Claude-User|Perplexity-User" /var/log/nginx/access.log \
| awk '{print $9}' | sort | uniq -c | sort -rnKeeping protection without collateral damage
Opening a door for agents raises a fair worry: will scrapers pretend to be agents to get through? They will try, which is why each step above insists on verification. Some further safeguards:
- Scope the allowance. Let verified agents skip the bot challenge, but keep them subject to rate limits, the WAF's attack signatures and login protection. An agent booking a table has no reason to send a SQL injection payload.
- Protect sensitive paths separately. Account pages, admin areas and APIs can keep stricter rules. Agents hand control back to the person when a login is needed, so they do not need a special route in.
- Refresh IP range lists automatically. Providers change their address ranges. A rule built from a list copied once will slowly start blocking the real agent while still allowing whoever inherits the old addresses. Fetch the published files on a schedule.
- Watch the numbers. A user-triggered fetcher visits because a person asked, so its volume should look like human demand. A sudden flood of requests claiming to be one agent is a sign of spoofing.
- Remember geo rules. Agent providers run their infrastructure in many countries. A rule that blocks all traffic from outside the UK will block most cloud agents even when the customer is in Leeds.
robots.txt is a separate matter. It tells crawlers what they may fetch, and some providers say their user-triggered fetchers do not follow it in the same way because a person asked for the page. It is a statement of your wishes, not an enforcement tool, so it cannot explain a 403 and it cannot stop a determined scraper.
Stopping it from happening again
- Add agent checks to change control. Any time someone changes CDN, WAF or security plugin settings, rerun the audit on one page from each main journey before closing the ticket.
- Remove emergency rules on a date. When a rule is added during an attack, give it an expiry date in its description and diary a review.
- Alert on sudden changes. A drop to zero in requests from agent user agents over a day is a useful alert that something upstream has changed.
- Keep a short policy note. Record which AI traffic you allow, which you block and why, so a new developer or agency does not "tidy up" the rules and undo it.
- Review after platform updates. CDNs regularly add new bot features and sometimes switch them on by default for new zones. Read the release notes for your security products.
1. How do I know if my site is blocking AI agents?
Run the AI Agent Compatibility audit, then test yourself from a cloud server using curl with an agent user agent such as ChatGPT-User. Look at the status code and at the body of the response, since a challenge page can come back with a 200. Finally, search your origin access log for the test. If no request appears there, something in front of the server, usually a CDN or cloud firewall, stopped it before it arrived.
2. If I block GPTBot in robots.txt, does that stop ChatGPT agents too?
No. GPTBot is the crawler OpenAI uses to gather training data, and a robots.txt rule for it has no bearing on ChatGPT-User, which fetches a page because a person asked, or on ChatGPT agent, which drives a browser. Most providers keep these separate on purpose so site owners can refuse training use while still being reachable by customers. Check each provider's documentation for the names it uses and what each one respects.
3. Is it safe to allow traffic by user agent?
Not on its own. Anyone can send a request that claims to be ChatGPT-User or Claude-User, so a rule that skips security for that name alone is an open door for scrapers. Combine the name with the provider's published IP ranges, use your CDN's verified bot flag, or check signed requests through Web Bot Auth. Keep rate limits and attack signatures applied even to verified agents, and allow only the challenge to be skipped.
4. What is Web Bot Auth?
Web Bot Auth is a proposal being developed at the IETF that lets an automated client prove who it is by signing its HTTP requests with a private key. The request carries signature headers and a Signature-Agent header that says where to find the public key. Your CDN or server checks the signature, which is very hard to fake. Cloudflare supports verifying these signatures, and some AI agent providers already sign their traffic, although adoption is still growing.
5. Why does the agent work from my office but not for customers?
Firewalls often trust your own IP address, your logged-in session cookie or traffic from your country, so tests from your desk skip rules everyone else meets. Cloud agents run from data centres, often in other countries, and arrive with no cookies. Rules based on IP reputation, hosting provider or geography hit them hard. Test from a small cloud server with no cookies to see what an agent actually receives from your site.
6. Should an agent be able to get past my CAPTCHA?
Reputable agents are built not to solve CAPTCHAs; they stop and ask the person to do it. That means a CAPTCHA shown to every visitor on the first page turns every agent task into a hand-over, and many will simply give up. Keep CAPTCHAs for moments of real risk, such as repeated failed logins or unusually fast form submissions, and use invisible or risk-based checks elsewhere so ordinary visits never see a puzzle.
7. Will allowing AI agents increase my server load?
User-triggered agents visit because one person asked for something, so their volume usually tracks human interest and is modest. Each visit loads a page much as a browser would. The traffic that causes heavy load is bulk crawling and scraping, which you can still limit or block separately. Keep per-IP rate limits on expensive paths such as search and basket, and watch your logs for a week after changing rules to see the real effect.
The AI Agent Compatibility audit tests your page for bot blocks, content hidden behind JavaScript, unlabelled forms and controls, navigation agents cannot follow and missing structured data, and links each finding to the guide that fixes it.
Open AI Agent Compatibility Fixes →