Fixing JavaScript That Crawlers Cannot Render
Some pages look perfect in a browser but reach search engines half-built: the title is a placeholder, the product grid is empty, pagination links are missing, or a "page not found" screen is served with a 200 status.
The cause is that the page depends on JavaScript to create things crawlers rely on, and rendering either fails, is incomplete or does not happen at all. This guide shows how to compare what the server sends with what a renderer builds, find the specific pieces that go missing, and fix each one so search engines see the same page your visitors do.
How search engines handle a JavaScript page
Google describes its process in three phases: crawling, rendering and indexing. Googlebot first fetches the URL and reads the HTML the server returns. Links and signals found in that raw HTML are available straight away. The page is then queued for rendering, where Google's Web Rendering Service loads it in an up-to-date version of Chromium, runs the scripts and takes the resulting DOM. Indexing works from that rendered result.
Several things can go wrong between those phases:
- Rendering is later than crawling. Google says most pages are rendered quickly, but the gap means new or changed content that only exists after scripts run can take longer to be picked up than content in the raw HTML.
- Rendering can fail. If a script throws an error, an API call times out, or a file the page needs is blocked by robots.txt, the renderer takes whatever DOM exists at that point.
- The renderer does not behave like a visitor. It does not click, type, grant permissions or keep cookies between page loads, and it does not scroll the way a person does. Content that waits for any of those may never appear.
- Not every crawler renders. Bing renders JavaScript too, but other search engines, social preview fetchers and many AI crawlers are widely reported to read only the raw HTML.
Symptoms that point to a rendering problem
| What you notice | Likely rendering cause |
|---|---|
| Search results show a generic title such as the app name on every page | The <title>
is set by script and the raw HTML has one shared placeholder. |
| Deep pages are "Discovered, currently not indexed" in Search Console | Links to them only exist after rendering, or are buttons with click handlers rather than links. |
| Missing pages are reported as indexed, or "Soft 404" appears | The app shows a not-found screen but the server returns 200 for every URL. |
| URL Inspection screenshot shows a spinner or blank area | A data request failed, timed out or was blocked during rendering. |
| Wrong canonical or noindex reported | Script changes the canonical or robots meta tag after load, so raw and rendered values disagree. |
| Only the first page of a listing is indexed | Further items load by infinite scroll or a "load more" button with no paginated URLs. |
Comparing raw and rendered HTML
The core diagnostic is simple: fetch the page without running scripts, fetch it again after rendering, and look at the differences. The JavaScript Checker loads the page in headless Chromium and compares the rendered page with what a crawler receives, which gives you that difference for a template in one pass. You can reproduce it by hand as well.
The raw HTML
curl -sL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
https://www.example.co.uk/shoes/trail-runner-2 -o raw.html
grep -o "<title>[^<]*" raw.html
grep -c "<a " raw.htmlUsing a crawler user agent also shows whether your server treats bots differently. View Source in the browser gives the same raw HTML for a quick look.
The rendered HTML
In DevTools, the Elements panel shows the live DOM after scripts have run. To save it, right-click
the <html> element and copy the outer HTML. In Search Console, URL Inspection gives
Google's own rendered version, described below.
What to compare
- Title, meta description, canonical and robots meta. These should be identical in both. If they only appear, or change, after rendering, fix them on the server.
- Main content. The heading, body text, product details and prices.
- Links. Count
<a href>elements in each. Navigation, pagination and category links should be in the raw version. - Structured data. JSON-LD added by script is usually processed after rendering, but server output is more dependable.
- Images. Check that product and article images have real
srcvalues, not only placeholder data attributes that a script swaps in.
Using URL Inspection properly
In Search Console, open URL Inspection, enter the address and choose Test live URL. When the test finishes, View tested page gives three tabs: the rendered HTML, a screenshot, and More info, which lists page resources that could not be loaded and JavaScript console messages.
- Rendered HTML is what matters for indexing. Search it for a sentence from the main content and for a few important links.
- The screenshot is only the top of the page, so do not judge missing lower content from it.
- Page resources shows files that failed or were blocked. An "Other error" on a script or API call during a live test can be a temporary limit, so test again before acting, but a file consistently marked as blocked by robots.txt needs fixing.
- JavaScript console messages show errors thrown during Google's render. An error here that you do not see in your own browser often points to code that relies on something the renderer does not provide, such as stored data or a granted permission.
The Rich Results Test uses the same rendering and does not need Search Console access, which makes it useful for pages on sites you do not have verified.
The fix, step by step
Step 1
Compare raw and rendered HTML for key templates
Run the JavaScript Checker on one URL of each main template and note what exists only after rendering. Confirm by fetching the raw HTML with curl and comparing it with the DOM in the Elements panel. Check the title, meta description, canonical, robots meta, main heading, body text, internal links and structured data. Make a list per template of every item missing from the raw HTML, because each one is a separate fix below.
Step 2
Check the rendered page in URL Inspection
For each template, run Test live URL in Search Console URL Inspection and open the tested page. Search the rendered HTML for a sentence of main content and for your pagination and category links. Open More info and write down any resources that failed to load and any console errors. Repeat the test once, because a single failed request can be temporary, but treat anything that fails every time as a real blocker.
Step 3
Unblock the files rendering depends on
Open robots.txt and remove any rule that blocks script bundles, stylesheets or the API paths
your pages call to fetch content, such as a blanket Disallow: /api/. Check that the firewall
or bot protection does not challenge Googlebot on those paths, and that the API responds quickly without
cookies. Fix any console error seen in the rendered test, since an uncaught error during rendering can
leave the page unfinished at that point.
Step 4
Turn script navigation into real links
Replace clickable elements that change the page with script, such as a div or button that
calls the router, with <a href> elements pointing to real URLs; your router can still
intercept the click for visitors. Stop using hash fragments such as #/products for distinct
pages and use the History API with normal paths. Give infinite scroll and load-more lists paginated URLs
that each work when loaded directly.
Step 5
Send correct status codes and head tags from the server
Make the server return 404 or 410 for URLs that do not exist, and 301 for moved pages, instead of returning 200 and letting the app draw an error screen. Output the unique title, meta description, canonical and any robots meta tag in the raw HTML for every route. Do not rely on script to change a noindex to index later, because Google may act on the first value it sees.
Step 6
Render essential content on the server
For each template, move the main heading, body text, product details, internal links and structured data into HTML the server sends, using static generation for pages that change rarely and server-side rendering for the rest. Most frameworks can do this per route. Keep client-side JavaScript for filters, baskets and other interaction. If a full change is not possible yet, prerender the most important routes at build time as an interim step.
Step 7
Watch coverage and logs after release
Re-run the checker and a URL Inspection live test on each template to confirm the raw and rendered HTML now match on the essentials. Request indexing for a few important pages. Over the following weeks, watch the Search Console page indexing report for fewer soft 404 and "Discovered, currently not indexed" entries, and check server logs to confirm Googlebot is fetching your deep pages and their resources successfully.
Patterns that break rendering, and their replacements
| Pattern | Why a crawler misses it | Replacement |
|---|---|---|
| Content loaded after a scroll event | The renderer does not scroll like a visitor, so the event may never fire. | Use IntersectionObserver for below-the-fold extras, and put
the main content in the HTML. |
| Tabs or accordions that fetch their content on click | No click happens during rendering. | Include the content in the HTML and show or hide it with CSS. |
| Content that depends on stored data or cookies | Storage and cookies are cleared between page loads during rendering. | Render a sensible default for a first-time visitor. |
| Content behind a geolocation or notification prompt | Permission requests are declined. | Show default content and let the visitor choose a location. |
| Links built only in click handlers | Crawlers follow href values, not script. | Real anchors with real URLs. |
| Hash-based routes | Everything after # is ignored when identifying a page. | Path-based routes via the History API. |
| Timers that wait several seconds before showing content | Rendering does not wait indefinitely for page activity. | Render immediately; animate afterwards if needed. |
Framework and platform notes
- React, Vue and Angular single-page apps. A default client-side build sends an almost empty body. Next.js, Nuxt, SvelteKit, Remix-style routers and Angular's server rendering can output full HTML per route; the React and Next.js guide covers the React side in depth.
- WordPress and Shopify. Core templates are server-rendered, so problems usually come from plugins or apps that inject reviews, related products or FAQs by script. Check those blocks in the raw HTML specifically.
- Static site generators. Output is already HTML, but client-side search pages and filtered listings built in the browser still need crawlable alternative URLs if you want them indexed.
- Dynamic rendering. Serving crawlers a separately prerendered copy still works, but Google calls it a workaround and it is easy for the two versions to drift apart. Use it only as a temporary bridge.
Stopping it from coming back
- Add a raw-HTML test to your pipeline. For each template, fetch the built page without JavaScript and assert that the title, canonical, main heading and a known link are present.
- Treat robots.txt as code. Review changes to it, and never block paths your pages need in order to render.
- Keep error monitoring on. A new uncaught error on page load can stop rendering, so alert on it.
- Re-run the checker after framework upgrades and after adding apps or plugins that inject content.
1. Does Google render JavaScript?
Yes. Google renders pages with an up-to-date version of Chromium and indexes the rendered result. However, rendering happens after the initial crawl, it depends on your scripts and data requests working without errors, and it will not click, type or grant permissions. So Google can see most script-built content, but anything that relies on interaction, cookies or blocked files can be missed. Content in the raw HTML avoids all of those risks and is seen straight away.
2. How do I see the HTML Google rendered for my page?
Use URL Inspection in Search Console. Enter the URL, choose Test live URL, then View tested page. The HTML tab shows Google's rendered DOM, the screenshot tab shows the top of the page, and More info lists resources that failed to load and JavaScript console messages. Search the rendered HTML for a sentence of your main content and for important links. The Rich Results Test uses the same rendering if you do not have Search Console access.
3. Why are my single-page app routes reported as soft 404?
Usually because the server returns a 200 status for every URL, and the app then draws a not-found message in the browser. Google sees a 200 page that looks like an error and flags it as a soft 404, or the reverse happens and thin error pages get indexed. Fix it on the server: return a real 404 for unknown routes, or at minimum add a noindex robots meta tag to the error view in the raw HTML.
4. Can Googlebot follow links created with JavaScript?
Googlebot follows links that are anchor elements with an href containing a real URL, whether the anchor is in the raw HTML or added by script before rendering finishes. It does not follow buttons, spans or divs that navigate through click handlers, and it does not treat hash fragments as separate pages. Links in the raw HTML are also found earlier, during the first crawl, so they are the most dependable way to get deep pages discovered.
5. Should I block my API endpoints in robots.txt?
Not if your pages call them to fetch content while rendering. When robots.txt blocks those requests, Google's renderer cannot load the data, and the page is indexed without it. If you want to keep API URLs out of search results, leave them crawlable and send an X-Robots-Tag noindex header on the API responses instead. Only block endpoints that no public page depends on, such as internal admin or account-only paths.
6. Is prerendering the same as server-side rendering?
Not quite. Prerendering usually means generating static HTML for routes ahead of time, either at build time or with a headless browser, and serving those files. Server-side rendering builds the HTML on each request. Both give crawlers complete HTML. Prerendering suits pages that change occasionally; server rendering suits pages with frequently changing data such as stock levels. Either is better than dynamic rendering, which serves bots a different version from people.
7. Do other search engines and AI crawlers render JavaScript?
Bing says it renders JavaScript, although it recommends not depending on it for important content. Many other crawlers, including a number of AI crawlers and the fetchers behind social and chat link previews, are widely reported to read only the raw HTML. Behaviour varies and changes over time, so the safe approach is to make sure titles, main content, links and structured data are in the HTML the server sends, and to use JavaScript for enhancements.
The JavaScript Checker loads your page in headless Chromium, compares the rendered page with what a crawler receives, captures console errors that can stop rendering, and lists every script with its size and execution time.
Open JavaScript Checker Fixes →