How to Fix Page Speed for AI Agents

Slow pages fail agents in ways slow pages don't fail humans. A user with a slow connection waits and gets the content. An agent has a budget — typically 5-10 seconds total — and at the deadline it returns empty-handed and tells the user "couldn't find that". Your slow page becomes "site doesn't exist" in the user's chat. This guide covers the speed targets and the fixes that hit them.

1. The agent speed budget

Crawler agents (GPTBot, ClaudeBot, PerplexityBot):
  TTFB    target  < 2000 ms
  Total   timeout 10-30 s

User-triggered (ChatGPT-User, Perplexity-User):
  TTFB    target  < 800 ms
  FCP     target  < 2500 ms
  Total   timeout 5-8 s

Mobile users on agent apps:
  TTFB    target  < 1000 ms
  FCP     target  < 3000 ms

2. Measure agent-perceived speed

# Run from US-East (Virginia) or EU-West (Ireland)
# Most AI infrastructure lives there

curl -w "@-" -o /dev/null -s -A "GPTBot/1.0" https://example.com/ <<'EOF'
   dns:    %{time_namelookup}s
   tcp:    %{time_connect}s
   tls:    %{time_appconnect}s
   ttfb:   %{time_starttransfer}s
   total:  %{time_total}s
   bytes:  %{size_download}
EOF

3. CDN edge cache — the single biggest win

# Cloudflare cache rule
URL: example.com/*
NOT URL: example.com/admin/* OR example.com/api/*

Settings:
  Cache Level: Cache Everything
  Edge TTL: 4 hours (or longer for static content)
  Browser TTL: 1 hour
  Bypass on Cookie: session_*

# Result for cached pages: TTFB drops from 600ms to 30-80ms globally
# Origin only hit on cache miss or authenticated request

For most marketing/content sites, this single change gets every agent fetch into target range. The CMS slowness becomes invisible — 99% of requests served by CDN.

4. Compression

# nginx Brotli + gzip
gzip on;
gzip_comp_level 6;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

brotli on;
brotli_comp_level 6;
brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

# Brotli typically 15-25% smaller than gzip
# Most agents accept Brotli (Accept-Encoding: br)

5. Minify HTML

# Strip whitespace, comments, redundant quotes
# Cuts 20-40% of HTML bytes typically

# Next.js: enabled by default in production
# Astro: enabled by default
# WordPress: use Autoptimize plugin or W3 Total Cache
# Custom: html-minifier-terser at build time

6. Defer non-critical JS

<!-- Bad: blocking JS in head -->
<script src="/analytics.js"></script>
<script src="/chat-widget.js"></script>

<!-- Good: defer or async -->
<script src="/analytics.js" defer></script>
<script src="/chat-widget.js" async></script>

<!-- Better: lazy-load on interaction -->
<script>
  // Only load chat widget when user shows intent
  document.addEventListener('scroll', loadChatOnce, { once: true, passive: true });
</script>

7. Server-render above-the-fold

See Fix JS-Only Content. SSR for critical content; CSR for interactive widgets. Agents get HTML immediately; humans get interactivity progressively.

8. Optional: lighter HTML response for agents

// Express middleware example
app.use((req, res, next) => {
  const ua = req.get('User-Agent') || '';
  if (/GPTBot|ClaudeBot|PerplexityBot|CCBot|OAI-SearchBot/i.test(ua)) {
    req.isAgent = true;
  }
  next();
});

// Route
app.get('/products/:id', async (req, res) => {
  const product = await db.getProduct(req.params.id);
  
  if (req.isAgent) {
    // Stripped HTML: just content + schema, no images, no JS, no CSS
    return res.render('product-agent', { product });
  }
  return res.render('product-full', { product });
});

// Agent version typically 80% smaller, parses in < 50ms

Allowed (not cloaking — same content, different format). For high-volume agent traffic, large reduction in compute cost too.

9. Eliminate origin latency spikes

• Database connection pooling — no fresh handshake per request
• Cache hot queries in Redis / Memcached
• Defer slow third-party APIs to client-side fetch
• Set explicit timeouts on external calls (don't let a slow API stall the response)
• Pre-warm caches on deploy — first request after deploy shouldn't be slow

10. Verify across agent types

UAS=("GPTBot/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0" "anthropic-ai/1.0" "OAI-SearchBot/1.0")
for ua in "${UAS[@]}"; do
  ttfb=$(curl -s -o /dev/null -A "$ua" -w "%{time_starttransfer}\n" https://example.com/)
  echo "$ttfb  $ua"
done

# Every TTFB should be < 0.8 (800ms)
# If 1+ exceeds, WAF challenge or origin congestion under those UAs