Agent Readiness in WordPress
WordPress sends complete HTML from the server, so on paper it suits AI crawlers well. In practice a typical WordPress site has five or six separate layers that each decide something about crawler access: core settings, the theme, an SEO plugin, a security plugin, a page builder and the host or CDN in front of it all.
When the Agent Readiness Audit reports a problem, the hard part is finding which layer caused it. This guide goes through each layer in the WordPress admin and on the server, with the settings and commands that locate and fix the fault.
- Six layers that decide what a crawler gets
- Visibility settings and robots.txt in WordPress
- Security plugins, hosts and CDNs
- Themes, page builders and content loaded later
- Structured data from plugins
- Author settings that leave AI systems guessing
- Duplicate URLs WordPress creates on its own
- The fix, step by step
- Keeping it fixed after updates
- Questions and answers
Six layers that decide what a crawler gets
On a hand-built static site, one file controls each thing a crawler meets. On WordPress, responsibility is spread across software written by different people, and they do not always know about each other. That is why an AI crawler can be blocked even though nobody on the team remembers blocking it. Before changing anything, work out which layer is responsible for the finding in front of you.
| Layer | What it controls for crawlers | Typical agent readiness fault |
|---|---|---|
| WordPress core | Site visibility setting, the virtual robots.txt, default canonical tags, author and attachment URLs | The "discourage search engines" box left ticked after launch |
| Theme | The HTML structure of every template, bylines, dates, headings | Bylines removed from the design, dates hidden, content wrapped in anonymous divs |
| SEO plugin | Title, meta robots, canonical, JSON-LD graph, sitemap, often a robots.txt editor | A robots.txt rule typed into the plugin editor and forgotten |
| Security plugin | Rate limits, bot blocking, country blocks, login protection | AI crawlers treated as unknown bots and refused with 403 or 503 |
| Page builder and widget plugins | How sections, tabs, sliders and listings are produced | Listings, reviews or pricing loaded by AJAX after the page opens |
| Host and CDN | Caching, firewall, bot management, redirects | An "AI bots" block switched on at the CDN, or a cached maintenance page |
The audit fetches your page with AI crawler user agents and reports what came back. If the response is a 403, a challenge page or a timeout, look at the last three layers. If the response is a normal 200 but content is missing, look at the theme and page builder. If the page arrives complete but the audit flags structured data, authorship or canonicals, the SEO plugin and core settings are where the fix lives.
Visibility settings and robots.txt in WordPress
The site visibility checkbox
Under Settings, Reading there is an option labelled "Discourage search engines from indexing this
site". It is meant for development copies, and it is ticked automatically on many staging environments.
On current versions of WordPress it adds a robots meta tag with noindex to every page. AI
search crawlers that honour that tag will leave your pages out, and the setting is easy to carry over
to the live site when a staging copy is pushed. Check it on the live site, not on the copy you work on.
You can check it from the command line if your host provides WP-CLI. A value of 0 means the box is ticked:
wp option get blog_public
wp option update blog_public 1Three possible sources for robots.txt
WordPress does not ship a robots.txt file. When nothing else exists, it answers requests for /robots.txt
with a virtual file generated on the fly, which normally just keeps crawlers out of /wp-admin/
while allowing admin-ajax.php. Plugins can add lines to that virtual file through a filter.
If someone uploads a real robots.txt to the web root, the web server serves that file instead,
and anything the plugins would have added is ignored.
That gives three places a rule can come from:
- A physical file in the web root, usually uploaded years ago by a previous developer.
- The robots.txt editor inside your SEO plugin, which writes either a physical file or additions to the virtual one depending on the plugin.
- Other plugins, such as some security and multilingual plugins, adding their own lines through the WordPress filter.
To find out which one you are looking at, check whether a physical file exists, then compare it with what the live domain returns:
ls -l /path/to/wordpress/robots.txt
curl -s https://www.example.co.uk/robots.txtIf there is no physical file but the output contains rules you do not recognise, a plugin is adding them. Deactivating plugins one at a time on a staging copy, and fetching robots.txt after each, finds the culprit quickly.
Which AI crawlers to allow is your decision. The AI crawler access guide explains the difference between training, search and user-requested crawlers and how to write rules for each.
Security plugins, hosts and CDNs
A security plugin sees an AI crawler as an unfamiliar automated visitor that requests many pages quickly. Depending on the settings, it may rate-limit it, show it a challenge, or block it outright. Robots.txt has no influence here: the plugin acts before WordPress decides what to serve.
Places to look, in order:
- The security plugin's live traffic or firewall log. Search for crawler names such
as
GPTBot,ClaudeBot,PerplexityBotorOAI-SearchBot. A blocked request is usually logged with the rule that caught it. - Rate limiting rules for crawlers. Some plugins allow generous limits for verified search engine crawlers but apply stricter limits to other bots. AI crawlers usually fall into the stricter group.
- Country blocking. If you block traffic from certain countries, remember that crawlers run from data centres that may be in any of them.
- The host's own firewall. Managed WordPress hosts often run a web application firewall you cannot see from the WordPress admin. Ask support whether AI crawler user agents are blocked or challenged.
- The CDN. If the site sits behind a CDN, check its bot management and AI crawler settings. Some offer a single switch that blocks known AI crawlers across the whole zone.
A quick test from your own computer shows whether a user agent rule is involved. Compare the status codes:
curl -s -o /dev/null -w "%{http_code}\n" https://www.example.co.uk/
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://www.example.co.uk/If the first returns 200 and the second returns 403, something is matching the user agent. If both return 200 but the audit still reports a block, the rule is probably based on IP address or behaviour, and only the logs will show it.
Themes, page builders and content loaded later
WordPress themes render posts and pages on the server, so the main text of an ordinary post is almost always in the HTML. The trouble starts with the parts added by widgets and builder modules. When the audit reports content that depends on JavaScript, these are the usual sources on WordPress:
| Element | How it is often built | What to change |
|---|---|---|
| Post grids and "load more" listings | First few items in HTML, the rest fetched through AJAX when a button is clicked or the page is scrolled | Use paginated archive pages with real links as well as, or instead of, infinite scroll |
| Review and testimonial widgets | Loaded from a third-party service by script | Choose an option that prints reviews into the page on the server, or quote key reviews in the text |
| Price tables and calculators | Built in JavaScript from a settings object | Keep a plain HTML table of the standard prices in the page |
| Tabs and accordions | Usually all content in HTML, hidden with CSS; some builders fetch each panel on click | Check the source: hidden but present is fine, fetched on click is not |
| Maps, store locators and booking forms | Embedded iframes or scripts | Write the address, areas served and opening hours as text beside the embed |
Caching and optimisation plugins sometimes add their own twist. Options that delay JavaScript until the user interacts do not hide server-rendered text, but lazy-load features applied to whole sections, rather than images, can leave a crawler with empty placeholders. If a section is missing only when the cache plugin is active, look at its lazy-load and "delay" settings.
The content extractability guide covers moving facts out of images and script-only widgets in more depth, and the semantic HTML guide covers theme markup.
Structured data from plugins
The major SEO plugins, including Yoast SEO and Rank Math, output a single JSON-LD block containing
a connected @graph: the website, the page, an article where relevant, the author as a Person
and the site owner as an Organization or Person. That is a good foundation.
Problems appear when other software adds its own markup on top.
- Theme schema. Some themes print their own Article or BreadcrumbList markup. The result is two descriptions of the same page that may disagree on the author, dates or headline.
- A second schema plugin. Dedicated schema plugins can duplicate what the SEO plugin already outputs unless you turn one of them off for that type.
- Shop and event plugins. WooCommerce and event plugins add Product or Event markup. These are useful, but check that they connect to, rather than repeat, the page the SEO plugin describes.
- Incomplete site settings. The graph can only name your organisation, logo and social profiles if you have filled them in. Blank settings produce a thin graph with no clear publisher.
To see everything a page outputs, view the source and search for application/ld+json.
More than one block is not wrong in itself, but two blocks describing the same article almost always
are. Decide which plugin owns each type, switch the others off for it in their settings, and complete
the organisation details in the SEO plugin. The WordPress
JSON-LD guide goes further into plugin-by-plugin settings.
Author settings that leave AI systems guessing
WordPress attaches every post to a user account, and the theme prints that user's public name as the byline. Three habits undermine it:
- Posting as "admin". If the byline says admin, editor or the company name, there is no person for a reader or an AI system to recognise. Create an account for each real writer, or at least set a real "Display name publicly as" value under Users, Profile.
- Empty biographies. The Biographical Info box on the profile screen feeds many themes' author boxes and archive pages. Left empty, the author archive is just a list of links.
- Author archives switched off. SEO plugins let you disable author archives, which makes sense on a single-author site. On a multi-author site it removes the only page that says who each writer is. If you disable them, provide an alternative, such as a team page with an anchor for each person, and link bylines to it.
Themes that hide the byline and date with CSS, or remove them from the template to look cleaner, cause the same problem from the other direction. The plugin still outputs an author in the JSON-LD, but nothing visible on the page backs it up. The author and trust signals guide covers what a complete author setup looks like.
Duplicate URLs WordPress creates on its own
WordPress produces several addresses for the same content without anyone asking it to. AI crawlers that fetch the extra copies waste their visits and may quote a version you did not intend. The main ones are:
- Attachment pages. Each uploaded image can have its own page. Since WordPress 6.4, new installations have these pages switched off, but older sites often still have them, and SEO plugins offer an option to redirect them to the file or parent post.
- Comment reply links. Older themes add
?replytocom=parameters to reply links, creating one copy of the post per comment. - Tag and category archives with one or two posts each, which repeat the post excerpts almost word for word.
- Plain and pretty permalinks. Links such as
/?p=123still work and redirect, but internal links that use them add a redirect to every visit. - www and non-www, http and https. These should be settled with redirects at the server or host, and the WordPress Address and Site Address settings should match the version you chose.
WordPress core already outputs a self-referencing canonical tag on single posts and pages, and SEO plugins extend it to archives. The canonical conflicts guide explains how to check that the canonical tag, redirects, sitemap and structured data all name the same URL.
The fix, step by step
Step 1
Confirm the site is public and visible
On the live site, open Settings, Reading and make sure Discourage search engines
from indexing this site is not ticked, or run wp option get blog_public and expect
1. Then check for maintenance, coming soon or membership plugins that show visitors a holding
page, and for password protection set by the host. Fetch the home page with curl and read the first lines
of the response to be sure it is your real content.
Step 2
Find which robots.txt is actually served
Look in the web root for a physical robots.txt, then fetch /robots.txt
from the live domain and compare the two. If no file exists, the output is WordPress's virtual file plus
any lines added by plugins. Open the robots.txt editor in your SEO plugin and read every rule. Remove
anything that blocks AI crawlers you actually want, keep deliberate choices, and purge the host or CDN
cache before testing again.
Step 3
Check the security plugin and CDN bot rules
Open the firewall or live traffic log in your security plugin and search for the crawler names the audit reported as blocked. Note which rule caught them: rate limiting, bad bot lists, country blocks or a challenge. Add an exception for the crawlers you want rather than loosening protection for everyone. Then check the CDN and host firewall for an AI crawler setting, and ask hosting support if you cannot see their rules.
Step 4
Test each template with scripts switched off
Pick one example of each template: a post, a page, a category archive, a product if you run WooCommerce, and the home page. Load each with JavaScript disabled in developer tools, or fetch it with curl and search for a sentence you know is there. Note every section that is empty or shows a loading message, trace it to the widget or builder module that produces it, and change that module to render on the server.
Step 5
Let one plugin own the structured data
View the source of a post and a page and count the application/ld+json blocks.
Where the theme, the SEO plugin and a schema plugin all describe the same article, choose one owner and
switch the others off in their settings. Fill in the organisation name, logo and social profile addresses
in the SEO plugin so the graph names a clear publisher, then check the result in a structured data validator.
Step 6
Give posts real author names and pages
Reassign posts published under admin or a generic account to the people who wrote them, using the bulk edit screen for posts. For each writer, set a public display name and fill in the Biographical Info box. Decide whether author archives stay on, and if you disable them, link bylines to a team page instead. Check the theme still prints the byline and published date where readers can see them.
Step 7
Close the duplicate URLs WordPress creates
Redirect attachment pages to their parent post or file, remove replytocom
links if your theme still makes them, and consider noindex or merging for thin tag archives. Confirm
that the WordPress Address and Site Address settings match the https host you redirect to. Finally, run
the Agent Readiness Audit again on the pages you tested in step 4, and keep that list for checks after
future updates.
Keeping it fixed after updates
WordPress sites change constantly through plugin and theme updates, and many agent readiness faults arrive silently with one of them. A little routine keeps them from lasting.
- Keep a short test list. Record one URL per template and a sentence from each. After major updates, fetch each with a crawler user agent and check the sentence is still there.
- Treat staging pushes with care. Many staging tools copy the database, including the site visibility setting. Check it straight after every push to live.
- Note who owns what. Write down which plugin produces robots.txt additions, which owns structured data, and which handles redirects. The next person to install a plugin will know what it might clash with.
- Review security settings after updates. Security plugins sometimes add new bot lists or stricter defaults in a new version.
- Watch the logs. A monthly search of the access log for the AI crawlers you allow, looking at the response codes, shows whether they are still getting through.
1. Does WordPress block AI crawlers by default?
No. The virtual robots.txt that WordPress generates only keeps crawlers out of the admin area and does not name any AI crawler. When AI crawlers are blocked on a WordPress site, the rule has almost always been added by someone: a line in an SEO plugin's robots.txt editor, a physical robots.txt file uploaded to the server, a security plugin setting, or a bot setting at the host or CDN. The audit result tells you which crawler is refused and how, which narrows the search.
2. Why does my robots.txt in the SEO plugin not match what the audit sees?
Usually because a physical robots.txt file exists in the web root. The web server returns a real file before WordPress runs, so the virtual file and any rules your plugins add to it are never used. Some hosts also cache robots.txt at the CDN. Check for the physical file over FTP or the file manager, delete or edit it deliberately, then purge the cache and fetch /robots.txt from the live domain to confirm what crawlers now receive.
3. Which WordPress security plugin settings affect AI crawlers?
The settings that matter are the ones that treat automated visitors differently from people: rate limits for crawlers, lists of blocked bots or user agents, challenge pages for suspicious traffic, and country blocking. AI crawlers are not always on the list of verified search engines that these plugins trust, so they can be caught by the stricter rules. Check the plugin's firewall log for the crawler names first, then add a specific exception for the ones you want rather than weakening the whole rule.
4. Do page builders like Elementor or Divi stop AI crawlers reading my pages?
Page builders render most of their content on the server, so the text in ordinary sections, headings and text modules is in the HTML crawlers receive. The risk comes from particular modules: listings that fetch more items when scrolled, third-party review widgets, some pricing and calculator modules, and anything that loads its content on click. Test each important page with JavaScript disabled, note what disappears, and replace or reconfigure only the modules that caused it.
5. Should I use Yoast SEO or Rank Math for structured data?
Either can produce a sound, connected JSON-LD graph for a typical site, and the choice matters less than making sure only one tool describes each page. Problems come from running an SEO plugin, a separate schema plugin and a theme that all output Article or Organisation markup with slightly different details. Pick the tool that already runs your titles and sitemap, switch the overlapping types off elsewhere, and fill in the organisation and author details it asks for.
6. Should I switch off WordPress author archives?
On a site where one person writes everything, author archives mostly duplicate the blog listing, so turning them off is reasonable. On a site with several writers, the author archive is often the only page that says who each person is and what they have written, which is useful to readers and to AI systems judging a source. If you turn them off, give each writer a section on a team or about page and link every byline to it.
7. Why does my WordPress site show a maintenance page to crawlers but not to me?
Maintenance and coming soon plugins usually let logged-in administrators see the real site while everyone else sees the holding page. You are logged in, so you see your content; crawlers are not, so they see the holding page. Log out or use a private window to check, fetch the page with curl, and remember that a cached copy of the maintenance page can also linger at the host or CDN after the plugin is switched off.
The Agent Readiness Audit requests your page with real AI crawler user agents, checks robots.txt and llms.txt, reviews JSON-LD coverage and semantic HTML, tests how much depends on JavaScript and measures how easily the content can be extracted.
Open Agent Readiness Fixes →