Agent Readiness in React and Next.js

Why React sites are often invisible to AI crawlers

A classic React single-page app sends the same tiny HTML file for every URL: a head, an empty <div id="root"></div> and a script bundle. The browser downloads the bundle, React starts, works out which page to show from the address, fetches the data and builds the page. Everything a visitor reads is created in those last steps.

Google renders JavaScript in a second pass, so client-rendered React pages can still appear in Google Search. Most AI crawlers do not do this. They read the HTML from the first response and extract text from it. For a client-rendered app, that text is often just the page title and a "You need to enable JavaScript" message.

Server rendering frameworks fix this in principle, because the server runs your components and sends real HTML. In practice, several React habits undo it:

• Fetching content inside useEffect, which only runs in the browser, so the server-rendered HTML shows a loading state.
• Loading components with server rendering switched off, which leaves an empty space in the HTML.
• Reading from window or localStorage to decide what to show, which forces content to wait for the browser.
• Navigating with click handlers instead of links, so crawlers cannot discover other pages.

What a crawler receives from each setup

The label on the box matters less than what arrives. Test the actual HTML response for each template, as described below, before deciding what to change.

Next.js specifics that trip people up

The use client boundary

In the App Router, components are server components unless a file starts with 'use client'. Client components are still rendered to HTML on the server for the first response, so marking a component as a client component does not hide its markup by itself. What hides content is fetching the data after the component mounts. This pattern leaves crawlers with the loading message:

'use client'
export default function Reviews({ productId }) {
  const [reviews, setReviews] = useState(null)
  useEffect(() => {
    fetch(`/api/reviews/${productId}`).then(r => r.json()).then(setReviews)
  }, [productId])
  if (!reviews) return <p>Loading reviews...</p>
  return <ReviewList items={reviews} />
}

The same content fetched in a server component arrives complete, and any interactive part can still be a small client component inside it:

export default async function Reviews({ productId }) {
  const reviews = await getReviews(productId)
  return <ReviewList items={reviews} />
}

Components loaded without server rendering

dynamic(() => import('./Specs'), { ssr: false }) tells Next.js to leave that component out of the server HTML. It is meant for things that genuinely cannot run on a server, such as a map library that needs window. Search your codebase for ssr: false and check none of those components hold text that matters.

Streaming and loading states

With loading.tsx or Suspense boundaries, Next.js sends the page frame first and streams slower parts later in the same response. The content is still in the HTML, but it comes at the end of the document. Keep the core text outside slow Suspense boundaries so it arrives in the first part of the response, and keep slow calls such as recommendations inside them.

Metadata and structured data

Use the Metadata API for titles, descriptions, canonical URLs and social tags, so they are part of the server response. Set metadataBase in the root layout so relative canonical paths become full URLs. JSON-LD is not part of the Metadata API; render it as a script tag in the page component, escaping the less-than character as the Next.js documentation recommends:

export default async function Page({ params }) {
  const { slug } = await params
  const guide = await getGuide(slug)
  const jsonLd = {
    '@context': 'https://schema.org',
    '@type': 'Article',
    headline: guide.title,
    dateModified: guide.updatedAt,
  }
  return (
    <article>
      <script type="application/ld+json"
        dangerouslySetInnerHTML={{ __html: JSON.stringify(jsonLd).replace(/</g, '\\u003c') }} />
      <h1>{guide.title}</h1>
      ...
    </article>
  )
}

Next.js also supports app/robots.ts and app/sitemap.ts, which generate those files at build or request time. If you use them, check the output at /robots.txt after each deploy, because a rule written for a staging environment can easily reach production.

Routing, links and status codes in single-page apps

Even with rendering fixed, single-page apps have three routing habits that confuse crawlers.

• Hash routes. Addresses such as example.co.uk/#/pricing are one URL to a crawler, because everything after the # is never sent to the server. Every "page" collapses into the home page. Switch to path-based routes such as /pricing, with the server configured to handle them.
• Click-handler navigation. A <div onClick={() => navigate('/pricing')}> works for a mouse but gives a crawler nothing to follow. Use your router's link component, such as next/link or React Router's Link, which render a normal <a href>.
• Everything returns 200. A client-only app answers every address, including ones that do not exist, with the same HTML and status 200, then shows "Page not found" by script. Crawlers store those as real pages. In Next.js, call notFound() when data is missing so the response is a 404; in a pre-rendered SPA, configure the host to serve a real 404 for unknown paths.

If you pre-render a Vite or similar app at build time rather than migrating, make sure every public route is in the pre-render list, including product and article pages generated from data. Routes left out fall back to the empty shell.

The fix, step by step

The rest of the audit on a React site

Rendering is the React-specific part. The other findings are fixed the same way on any stack, and each has its own guide:

• Crawler access. Hosting platforms and CDNs in front of React apps often have bot protection or AI-crawler blocking settings. See fixing AI crawler robots.txt blocks.
• Structure. Component-heavy code tends towards deep div nesting. Fixing non-semantic HTML covers landmarks and headings, including passing heading levels into reusable components.
• Extractability. Content inside tabs, accordions and carousels that only load on interaction; see fixing poor content extractability.
• Structured data and URLs. JSON-LD coverage and canonical conflicts, which in React apps often come from trailing-slash settings and query-string variants.
• Authorship and llms.txt. Author and trust signals and the optional llms.txt file, which in Next.js can be a static file in the public folder or a route handler.

Speed matters too once the HTML is complete: a server-rendered page with slow data calls and no caching can still be too slow for a crawler. See fixing slow rendering for AI crawlers.

Keeping it right as the app grows

1. Agree a team rule: public content is fetched on the server; browser-side fetching is for personal or interactive data only.
2. Flag new uses of ssr: false and new 'use client' files in code review and ask what they render.
3. Keep the raw-HTML test from step 7 in the pipeline, and extend it when you add a template.
4. After framework upgrades, re-check route rendering modes, because defaults for caching and static generation have changed between major Next.js versions.
5. Re-run the Agent Readiness Audit after hosting, CDN or framework changes.