Fixing Poor Content Extractability
An AI crawler can receive your page perfectly and still come away with very little. The price list is a picture, the comparison table is forty nested divs, the delivery details only load when someone clicks a tab, and the opening hours sit inside a map.
A person sees everything; a text extractor sees a menu, a cookie notice and some fragments. This guide shows how to find the facts that are locked inside containers a machine cannot read, and how to move them into plain HTML text that any crawler, screen reader or reader mode can pick up with its meaning intact.
Fetched is not the same as readable
Extraction is the step after the crawler gets your HTML. Software has to strip away navigation, banners and footers, keep the main content, and turn it into text that still means what you meant. Anything that is not text in the HTML, or whose meaning depends on how it looks, tends to fall out at this stage.
The Agent Readiness Audit measures how easily the main content can be extracted. A poor result usually has one of three causes:
- The fact is not text. It is an image, a chart drawn on a canvas, a file in an embedded viewer or an icon.
- The fact is not in the HTML yet. It is fetched by a script when someone clicks, scrolls or hovers.
- The fact loses its meaning without layout. A grid of divs looks like a table, but once the styling is gone the numbers are no longer attached to their labels.
This guide deals with those three. Whether the page is built on the server at all is covered in the slow rendering guide, and wrapping the page in the right landmark elements is covered in the semantic HTML guide.
Where facts get locked away
| Container | What a text extractor gets | Better approach |
|---|---|---|
| Price list, menu or timetable saved as an image | The alt text at best, often just a file name | Write the list as HTML; keep the image as decoration if you like it |
| Infographic with key figures | Nothing from inside the graphic | Put the figures in a paragraph or table next to it |
Chart drawn on <canvas> | Nothing; canvas holds pixels, not text | Add a data table or a sentence stating the main values |
| PDF brochure or spec sheet | A separate file that may never be fetched; scanned PDFs contain no text at all | Publish the key content as an HTML page and offer the PDF as a download |
| Map or booking widget in an iframe | Content that belongs to another address, not your page | Write the address, area and hours in the page text |
| Ticks, crosses and stars as icons | Empty elements or icon-font characters with no meaning | Add visible or visually hidden text such as "Included" or "4 out of 5" |
Text added with CSS content: | Usually nothing; generated content is not part of the HTML | Put meaningful words in the markup, use CSS only for decoration |
| Carousel slides loaded one at a time by script | The first slide only, or none | Output all slides in the HTML, or list the key points below |
Tables that keep numbers attached to labels
Comparison grids, price tiers and specification sheets are often built from rows of divs styled to look like a table. On screen they are clear. Once a machine strips the styling, it gets a flat run of words and numbers with nothing saying which value belongs to which column. A real HTML table carries that relationship in the markup itself.
A layout that looks like a table but is not one:
<div class="plans">
<div class="col"><div>Basic</div><div>£12</div><div><i class="icon-tick"></i></div></div>
<div class="col"><div>Plus</div><div>£20</div><div><i class="icon-tick"></i></div></div>
</div>The same information as a table, with a caption, header cells and words instead of icons:
<table>
<caption>Window cleaning plans, prices per visit, September 2026</caption>
<thead>
<tr><th scope="col">Plan</th><th scope="col">Price per visit</th><th scope="col">Conservatory roof</th></tr>
</thead>
<tbody>
<tr><th scope="row">Basic</th><td>£12</td><td>Not included</td></tr>
<tr><th scope="row">Plus</th><td>£20</td><td>Included</td></tr>
</tbody>
</table>Three details do most of the work. The <caption> says what the table is and when
it applies. The <th> cells with scope tie every value to both its row
and column. And the words "Included" and "Not included" replace icons that meant nothing without their
colour. You can still make the table look like cards on a phone with CSS; the structure stays underneath.
Tabs, accordions and "read more"
Product specifications, delivery terms and FAQs often sit behind tabs or accordions. Whether a crawler can read them depends entirely on how the panel is built, not on whether it is open.
- Hidden but present. All panels are in the HTML, and CSS or a script only controls which one is visible. Crawlers that read the source get everything. This is fine.
- Fetched on demand. The panel is empty in the HTML and a script requests its content when someone clicks. A crawler that does not click, and most do not, gets nothing.
- Truncated. A "read more" link that loads the rest of a description by script has the same problem as a fetched tab.
To tell which you have, view the page source rather than the inspector, and search for a sentence from a closed panel. If it is there, the panel is fine. If it only appears in the inspector after you click, it is loaded on demand.
The simplest robust pattern for an accordion is the native disclosure element, which needs no script at all and keeps the content in the HTML:
<details>
<summary>Delivery and returns</summary>
<p>We deliver to mainland UK addresses within three working days.
Unused items can be returned within 30 days of delivery.</p>
</details>For tabs, keep every panel in the markup and let the script switch visibility. If a panel is genuinely too heavy to load up front, such as hundreds of reviews, include a short summary in the HTML and load the full list afterwards.
Passages that still make sense on their own
Extraction systems and answer engines often work with parts of a page rather than the whole thing: a paragraph, a list, a table row. A passage that relies on the one before it loses its meaning when lifted out. Small writing habits make each passage self-sufficient without making the page read like a list of soundbites.
- Name the subject. Write "The Plus plan includes conservatory roofs" rather than "It also covers those".
- Put units and dates with numbers. "£20 per visit, from September 2026" survives being quoted; "20" does not.
- Avoid pointers to the layout. "See the table on the right" and "click the tab above for prices" mean nothing once the layout is gone. Say what the reader will find.
- Lead with the answer. Under a heading that asks or implies a question, put the direct answer in the first sentence and the detail after it.
- Keep one idea per paragraph. Long paragraphs that switch topic halfway make it hard to extract either idea cleanly.
None of this means shortening your content or removing nuance. It means each block of text carries its own context.
Testing what an extractor gets
Reader mode
Firefox Reader View, and the similar modes in other browsers, use heuristics to pick out the main content. Open the page in reader mode and compare it with the normal view. Missing tables, missing sections, or a reader view that shows only the cookie notice and footer are strong warnings.
Search the raw source for each key fact
Fetch the page as a crawler would and search for the facts you listed. If a price or opening time does not appear, it is not in the text a non-rendering crawler receives:
curl -s -A "ClaudeBot" https://www.example.co.uk/plans/ | grep -o "£[0-9]*" | sort -uTurn off images and styles
In developer tools, disable CSS and block images, then read the page from top to bottom. What remains is close to what an extractor works with. Numbers floating without labels point to div-built tables; gaps where pictures were point to text in images.
Select all and paste
Select the whole page, copy it and paste into a plain text editor. Crude, but quick: whatever does not survive the paste probably does not survive extraction either.
The fix, step by step
Step 1
List the facts each page must deliver
For each important template, write down the facts a customer or an answer engine would most want from it: prices, sizes, specifications, delivery terms, opening hours, service areas, contact details, eligibility rules. Keep the list short and concrete. This is your test set. A page passes when every one of these facts can be found as plain text in the HTML the server sends, next to a label that says what it is.
Step 2
Find where each fact actually lives
For every fact on the list, check the page source and note its container: plain text, image, PDF, iframe, canvas chart, icon, script-loaded panel or CSS-generated text. Use reader mode and a curl fetch alongside the source. Anything that is not plain text in the source goes on a fix list, grouped by template, because a fault in a template usually affects every page built from it.
Step 3
Replace pictures of text with real text
Rewrite price lists, menus, timetables and key infographic figures as HTML paragraphs, lists or tables. Keep the image if it helps the design, with short alt text describing it. For PDFs, publish the essential content as an HTML page and link the PDF as a download. Beside maps and booking widgets, write the address, service area and opening hours in ordinary text so they belong to your page.
Step 4
Rebuild data layouts as HTML tables
Convert
comparison grids, price tiers and specification layouts built from divs into real <table>
markup with a <caption>, header cells and scope attributes. Replace tick,
cross and star icons with words or add hidden text that carries the meaning. Keep the visual design with
CSS, including stacked cards on small screens, so readers see no change while the structure underneath
becomes readable.
Step 5
Put tab and accordion content in the source
For each tab, accordion and read more link, confirm whether the hidden text is already
in the page source. Where it is fetched on click, change the component so all panels render into the
HTML and the script only toggles visibility, or use <details> and <summary>.
For very large lists such as reviews, put a summary in the HTML and load the rest afterwards.
Step 6
Make key passages stand on their own
Reread the paragraphs that hold your listed facts. Name the product or service instead of relying on it or this, put units, currency and dates next to every figure, and remove phrases that point at the layout such as see the table on the right. Where a heading asks a question, answer it in the first sentence underneath. Keep each paragraph to a single idea so it can be quoted accurately.
Step 7
Retest with reader mode and a raw fetch
Open each fixed template in reader mode and confirm the tables and former tab content now appear. Fetch it with curl using an AI crawler user agent and search for every fact on your list. Disable CSS and images once more and check that numbers sit beside their labels. Then run the Agent Readiness Audit again and compare the extractability result with the one you started from.
Reducing the noise around the content
Extraction is also harder when the main content is buried. A page where the source contains several thousand words of mega-menu, cookie text, pop-up offers and repeated calls to action before the first paragraph of real content gives the extractor more to discard, and more chances to keep the wrong part.
- Keep boilerplate compact. Large navigation menus repeated in the header, a sidebar and the footer triple the noise on every page.
- Load consent banners with care. Cookie and newsletter overlays are best added in a way that does not put their text ahead of the main content in the HTML.
- Do not duplicate the page text. Some themes print the same content twice, once for desktop and once for mobile, and hide one with CSS. Extractors may keep both copies or pick the wrong one.
- Mark the main area. Wrapping the content in
<main>and<article>gives extractors a clear boundary.
These changes also help people using screen readers, who meet the same obstacles in the same order. Extractable content and accessible content are very largely the same thing.
1. What does content extractability mean?
It describes how easily software can take the page it fetched and pull out the main content as meaningful text. A page can be delivered perfectly and still extract badly if its facts sit in images, embedded files, icons or panels that load by script, or if tables are built from styled divs that lose their structure without CSS. Good extractability means every important fact is present as plain text, with its label beside it, inside a clearly marked main content area.
2. Can AI crawlers read text inside images?
Some AI systems can read text in images when an image is handed to them directly, but you should not count on a crawler downloading and analysing every picture on your page while it gathers text. Most extraction works on the HTML. A price list saved as a JPG is therefore very likely to be missed, and even when it is read, the result can be less reliable than text. Write the facts in HTML and keep the image as decoration.
3. Is content hidden in an accordion ignored?
Not if it is in the HTML. Content that is present in the source and simply hidden with CSS until someone clicks can be read by any crawler that parses the page. The problem is accordions and tabs that are empty in the source and fetch their text with a script when opened, because crawlers generally do not click. View the page source, not the developer tools inspector, and search for a sentence from a closed panel to see which kind you have.
4. Are PDFs a problem for AI crawlers?
A text-based PDF can be read, but it is a separate file at a separate address, so it depends on a crawler choosing to fetch it, and it is often not linked in a way that makes its importance clear. A scanned PDF contains only pictures of pages, with no text at all unless it has been through character recognition. For important information such as prices, terms and specifications, publish an HTML version and offer the PDF as an optional download.
5. How does the reader mode test help?
Reader modes in browsers such as Firefox use their own rules to find the main content and strip everything else, which is similar to what many extraction tools do. If reader mode shows your tables, headings and full text, extraction is probably in good shape. If it shows only the cookie banner, a fragment of the page, or numbers with no labels, an automated extractor is likely to struggle in the same way. It is a quick, free check to run before and after a fix.
6. Will changing div layouts into tables break my mobile design?
It does not have to. HTML tables can be restyled with CSS so that on a phone each row appears as a stacked card, with the header text shown beside each value. The structure underneath stays a proper table with header cells, which is what machines and screen readers need. Test on a real phone and with a screen reader after the change, and keep the table genuinely for data rather than for arranging parts of the page.
7. Is this the same as making my site accessible?
There is a large overlap. Screen readers and text extractors both depend on meaningful text in the HTML, real tables with headers, words rather than icons for meaning, and content that exists without a click. Fixing extractability therefore usually improves accessibility too, and the reverse is also true. Accessibility covers more, such as keyboard use, focus and colour contrast, so an accessibility review is still worthwhile even after these fixes.
The Agent Readiness Audit requests your page with real AI crawler user agents, checks robots.txt and llms.txt, reviews JSON-LD coverage and semantic HTML, tests how much depends on JavaScript and measures how easily the content can be extracted.
Open Agent Readiness Fixes →