How to Fix Agent Compatibility in WordPress for AI Crawlers

AI agents (GPTBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot) need to crawl your WordPress site to learn from it. By default, WordPress sites block or rate-limit several AI agents via security plugins or robots.txt. This guide covers WordPress-specific configuration to grant the agents access while keeping malicious bots out. Combine with the agent compat guide and robots.txt fixes.

Step 1: Audit current bot access in robots.txt

Visit yoursite.com/robots.txt. Look for Disallow rules that block AI agents by user-agent. WordPress generates a virtual robots.txt by default which is permissive; SEO plugins (Yoast, Rank Math, AIOSEO) may add restrictions.

Step 2: Choose your AI bot policy

Decide which AI agents you allow. Mainstream stack: allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot. Block training-only bots if you have copyright concerns. Document the decision in robots.txt comments.

Step 3: Edit robots.txt via SEO plugin

Yoast SEO → Tools → File editor → robots.txt. Rank Math → General Settings → Edit robots.txt. AIOSEO → Tools → Robots.txt editor. Add explicit Allow rules for the agents you welcome.

Step 4: Audit security plugin block lists

Wordfence, Sucuri, iThemes Security: check their bot-blocking rules. They often default to blocking 'aggressive crawlers' which includes AI agents. Whitelist the AI user-agents explicitly.

Step 5: Remove or tune rate limits

Cloudflare Bot Fight Mode, Wordfence rate limiting, and most managed WordPress hosts (WP Engine, Kinsta) rate-limit non-Googlebot crawlers. Whitelist AI agent IP ranges or user-agents in the relevant dashboard.

Step 6: Expose semantic markup

AI agents prefer clean HTML5 semantic markup (article, section, nav, main). Confirm your theme uses these. Avoid divs-only output from older page builders.

Step 7: Verify access

Use a user-agent switcher in Chrome DevTools. Set UA to 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot' and reload your site. Check server logs for the request reaching the origin.

Frequently Asked Questions

Should I block GPTBot on WordPress?

Depends on your business model. If you're a publisher whose value is original content and you don't want it absorbed into training data, block GPTBot. If you're a service/product business hoping to be cited in ChatGPT responses, allow it. There's no right answer — but make a deliberate choice rather than defaulting.

Why is Wordfence blocking AI bots?

Wordfence's default 'live traffic' view treats any non-mainstream crawler as suspicious. By default it doesn't actively block, but it triggers rate limiting on AI agents which behave differently from Googlebot. Add AI agent user-agents to Wordfence's whitelist under Wordfence → Tools → Diagnostics.

Does the WordPress llms.txt plugin help?

Yes if you want to publish a curated knowledge base for AI agents. llms.txt is a separate file (yoursite.com/llms.txt) that points AI agents at the most valuable content. Plugins like 'LLMs.txt for WordPress' generate it automatically from your pages. Combine with robots.txt access.

Will Cloudflare block AI agents?

Cloudflare added a one-click AI bot blocker in 2024. It's opt-in via Security → Settings → Bots → Block AI Scrapers. Check this isn't on. If you want partial blocking (e.g. only block training bots, allow citation bots), configure per-bot rules in Cloudflare WAF.

How do I see which AI bots are actually visiting my site?

Server access logs (cPanel → Raw Access, or WP Engine → Live logs). Search for user agents containing 'GPTBot', 'PerplexityBot', 'ClaudeBot', 'CCBot', 'anthropic-ai', 'Google-Extended'. Or install a logging plugin like WP Activity Log with bot-specific tracking.

🤖 Check AI agent compatibility

Verify GPTBot, PerplexityBot and ClaudeBot can crawl your site.

Run Agent Compat Audit →