How to Fix JS-Only Content for Agents

AI agents using lightweight HTTP fetchers (GPTBot, ClaudeBot, PerplexityBot) don't execute JavaScript. Even agents that do (ChatGPT-User browsing) have aggressive timeouts. If your critical content renders after JS executes, agents see an empty shell with loading spinners. The fix: ensure critical content is in the initial HTML response — via server-side rendering, static generation, or pre-rendering for bot user agents.

1. Audit current state

curl -s https://example.com/your-page | grep -i "your-main-content-keyword"

# Or save and inspect:
curl -s https://example.com/your-page > raw.html
wc -l raw.html
grep -c "
" raw.html

# If content keyword absent and <p> count is tiny, page is JS-rendered

curl -s https://example.com/ | head -50

# Bad sign: <div id="root"></div> and nothing else
# Good sign: full content visible in <main>, <article>, <section>

2. Rendering strategies

Strategy 1: Static Site Generation (SSG)

Best for: content sites, blogs, docs, marketing pages. Content rarely changes per request.

# Astro
npm create astro@latest

# Next.js (SSG)
export async function generateStaticParams() {
  const posts = await getPosts();
  return posts.map(post => ({ slug: post.slug }));
}

# Hugo, Eleventy, Jekyll — all SSG

Strategy 2: Server-Side Rendering (SSR)

Best for: dynamic content per request, personalised pages, dashboards with public landing.

// Next.js App Router — SSR by default
export default async function Page({ params }) {
  const data = await fetchData(params.id);
  return <article>{data.content}</article>;
}

// Nuxt
<script setup>
const { data } = await useFetch(`/api/posts/${slug}`);
</script>
<template>
  <article>{{ data.content }}</article>
</template>

Strategy 3: Pre-rendering for bots (existing SPA)

Best for: existing CSR React/Vue SPAs that can't be migrated. Serves static HTML to bots, regular SPA to humans.

# Prerender.io middleware (Express)
const prerender = require('prerender-node');
app.use(prerender.set('prerenderToken', 'YOUR_TOKEN'));

# Or self-hosted with Rendertron
# Nginx config detects bot UA and proxies to renderer
location / {
  if ($http_user_agent ~* "GPTBot|ClaudeBot|PerplexityBot|Googlebot") {
    proxy_pass http://localhost:3001/render/$scheme://$host$request_uri;
  }
  try_files $uri /index.html;
}

Strategy 4: Hybrid — critical content SSR, rest CSR

Best for: complex apps where full SSR is expensive. SSR the parts agents need.

// Next.js: SSR the public-facing pages, CSR the app
// /blog/[slug] — server component (SSR)
// /app/dashboard — client component (CSR)

// Astro: islands architecture — SSR by default, hydrate only what needs JS
<Header />       <!-- pure HTML -->
<ArticleBody /> <!-- pure HTML -->
<CommentSystem client:visible /> <!-- hydrates on view -->

3. Migration: React SPA to Next.js

# 1. Install Next.js alongside existing app
npx create-next-app@latest my-app-next

# 2. Move pages to app/ directory as server components
# Old: src/pages/About.jsx (CSR)
# New: app/about/page.tsx (SSR)

# 3. Convert client-only hooks/state
# - useState, useEffect → require 'use client' directive
# - Keep top-level page as server component
# - Mark interactive subcomponents 'use client'

# 4. Replace API calls
# Old: fetch from useEffect
# New: await fetch in async server component

# 5. Verify with curl after deploy
curl -s https://example.com/about | grep "Welcome to About"
# Should show content in raw HTML

4. What MUST be in initial HTML

• Main page content (article body, product description)
• Page title and meta description
• Heading hierarchy (h1-h6)
• Structured data (JSON-LD)
• Primary navigation links
• Canonical and hreflang tags
• Open Graph and Twitter card metadata

5. What can stay JS-rendered

• Interactive widgets (calculators, configurators)
• Personalised recommendations (logged-in user)
• Live data feeds, chat windows
• Analytics, A/B test variants
• Comment systems, social embeds

6. Verify after migration

for ua in "GPTBot/1.0" "ClaudeBot/1.0" "Googlebot/2.1"; do
  echo "=== $ua ==="
  curl -s -A "$ua" https://example.com/your-page | \
    grep -o "your main content phrase"
done

# Each should output the phrase. If empty, content still JS-only.