Fixing Slow Rendering for AI Crawlers

Why speed and rendering matter more for AI crawlers

A person on a slow connection waits, watches a spinner and eventually reads the page. A crawler works differently. It has a queue of URLs to get through, it sets a limit on how long each request may take, and when a request runs past that limit it moves on. None of the major AI providers publish their exact limits, so there is no official number to aim for, but the logic is simple: the longer your server takes, the more likely a fetch is to be abandoned or deprioritised.

The second difference is rendering. A browser downloads your HTML, then your scripts, then runs those scripts to build the page. Google's crawlers do something similar, in a later rendering pass. Most AI crawlers do not. They keep the HTML from the first response and extract text from it. If your product descriptions, article bodies or prices only appear once a script has run and called an API, the crawler's copy of the page does not contain them.

So "slow rendering" in the audit covers two separate faults that often travel together:

• A slow first response. The server takes a long time to start sending the HTML, usually because every request builds the page from scratch with database queries and nothing is cached.
• Content that arrives late. The HTML arrives quickly but is incomplete, and the real content only appears after scripts run. For a crawler that does not run scripts, "late" means "never".

Fixing the first without the second gives crawlers an empty page faster. Fixing the second without the first gives them a complete page they may not wait for. You need both.

Measuring what a crawler waits for

Browser tools report a lot of timings that crawlers never experience, such as image decoding and layout. For a crawler, measure the plain HTTP request. curl can print a timing breakdown for a single fetch:

curl -s -o page.html -A "ClaudeBot" \
  -w "status %{http_code}\nconnect %{time_connect}s\ntls %{time_appconnect}s\nfirst byte %{time_starttransfer}s\ntotal %{time_total}s\nbytes %{size_download}\n" \
  https://www.example.co.uk/guides/boiler-servicing/

Read the output like this:

Run it several times. The first request after a quiet period is often slow because nothing is cached; the second may be fast. Crawlers frequently request pages that nobody has visited recently, so the slow, uncached figure is the one that reflects their experience. Adding a random query string, such as ?nocache=123, often forces a miss on simple caches and shows you the origin time.

Your own connection is not the crawler's. AI providers run their crawlers from their own infrastructure, often in data centres far from a UK host. If your server sits in one UK data centre with no CDN, a crawler's connection time will be higher than yours. An online testing service that lets you pick a test location and a custom user agent gives a fairer picture than a test from your desk.

Spotting an empty or half-built page

The fastest test is to look at page.html from the curl command above. Open it in a text editor and search for a distinctive sentence from the middle of the page. Then look at what the body contains. These patterns mean a crawler gets little or nothing:

• A body that is mostly one element such as <div id="root"></div> or <div id="app"></div> followed by script tags. This is a client-rendered single-page app.
• Headings and layout present, but placeholder text such as "Loading..." or skeleton blocks where the content should be. The template is rendered on the server but the data is fetched in the browser.
• The article text present, but prices, stock, reviews or specifications missing. Those parts come from a widget or app that loads after the page.
• All the text present inside a large <script> block as JSON, but not in the visible HTML. Some frameworks ship page data this way for hydration. A crawler that only extracts visible text will not use it.

In a browser, you can confirm the same thing by turning JavaScript off in developer tools and reloading. What is left is roughly what a non-rendering crawler receives. Do not rely on the Elements panel with scripts on: it shows the page after JavaScript has built it, which is exactly what the crawler does not see.

Rendering options compared

If the content is added by scripts, the lasting fix is to change where the HTML is built. There are four common approaches, and most modern frameworks support more than one.

Static generation is usually the best choice for content pages because it removes both problems at once: the HTML is complete and it is a file, so it is served almost instantly. Server-side rendering with a cache in front comes close. Pre-rendering is a workaround; Google has described serving a separate rendered version to bots, which it calls dynamic rendering, as a stopgap rather than a long-term solution, and the same reasoning applies to AI crawlers. The framework details for React and Next.js are in the React and Next.js agent readiness guide.

The fix, step by step

Where the slowness usually comes from

WordPress and other PHP CMSs

Without page caching, every request runs PHP, loads plugins and queries the database. A caching plugin or host-level page cache turns that into a file read. Most WordPress pages are rendered on the server already, so the rendering half of the problem is usually limited to page-builder widgets, review plugins and sliders that load content with scripts. The WordPress agent readiness guide covers the settings.

Shopify and hosted platforms

The platform runs the servers and a CDN, so first-byte time is largely out of your hands and is usually reasonable. The rendering problems come from apps that inject reviews, size guides, FAQs and bundles with scripts. See the Shopify agent readiness guide.

JavaScript frameworks

This is where empty HTML is most common. A single-page app built with a client-only template sends a shell. A framework that supports server rendering can still produce empty pages if the data is fetched inside browser-only code. Fixing it is a code change, not a setting.

Headless setups

A headless CMS feeding a front end built separately adds a network hop on every uncached render. Cache API responses on the server side as well as the HTML, and prefer building pages at deploy time so the CMS is not called on each crawler request.

Pitfalls and edge cases

• Serving bots something different. If you pre-render for crawlers, the saved HTML must match what people see. Adding extra keyword text or hiding content only in the bot version is cloaking, and search engines treat it as spam. The safest approach is one HTML response for everyone.
• Bot protection that adds delay. Some firewall modes hold every unfamiliar client for a JavaScript challenge. A crawler cannot pass it, so the "slow page" is really a blocked page. The crawler access guide covers allowing verified crawlers.
• Cache keyed on user agent. If your cache varies by the full user-agent string, every crawler version gets its own cold cache and is always served slowly. Vary on device type at most.
• Streaming responses. Some frameworks stream HTML in chunks, sending the page frame first and the slower parts later in the same response. The text still arrives in the HTML, but a crawler with a short time limit may cut the stream off before the last chunk. Keep the main content in the first part of the stream.
• Soft errors. A client-rendered app often answers every URL with status 200 and shows "not found" by script. Crawlers then store an error page as if it were content. Make genuinely missing pages return 404 from the server.
• Very large documents. Google documents that Googlebot only processes the first 15 MB of an HTML file. AI crawlers publish no such figure, and some may stop much sooner. Keep documents small and put the main content early in the source.

Keeping it fast after the fix

Speed and rendering regress quietly, because the site still looks fine in a browser. Build a few checks into your routine:

1. Keep a short list of test URLs, one per template, and re-run the curl timing and raw-HTML text search after every deployment, theme update or new plugin or app.
2. Add an automated check to your build or monitoring that fetches each test URL without running scripts and fails if a known phrase is missing.
3. Watch the proportion of cache hits in your CDN dashboard. A sudden drop usually means a new cookie or header is making every request look private.
4. Review crawler entries in your access logs monthly for rising response times or error codes.
5. Re-run the Agent Readiness Audit after platform or hosting changes, which is when these problems most often start.

For a deeper look at server response time in general, see the TTFB fix guide. For the separate problem of facts hidden in tabs, images and PDFs even when the HTML is complete, see fixing poor content extractability.