Ecommerce Crawl Example: 38k Faceted URLs, 1,820 Zombie OOS Products
This example shows the aiwebpageseo Ecommerce Crawl for shop.example.com. 12,480 products crawled, 94% Product schema validation pass, 1,820 out-of-stock products without proper OOS signalling, 38,420 faceted URL variants exploding crawl budget, and 340 orphan products with zero internal links.
| Section | URLs | 200 OK | Schema valid | Status |
|---|---|---|---|---|
| /category/* (PLPs) | 240 | 240 | 100% | ✅ Excellent |
| /product/* (PDPs) | 12,480 | 12,420 | 94% | ✅ Strong |
| /collections/* | 87 | 87 | 92% | ✅ Good |
| /blog/* (editorial) | 620 | 620 | 100% | ✅ Good |
| /?facet=* (faceted) | 38,420 | 38,420 | N/A | 🔴 Crawl explosion |
38,420 faceted URLs indexable — crawl budget explosion
Filtering by colour + size + price + sort generates 38k+ indexable URL variants from 240 base categories. Google wastes crawl budget; ranking signal diluted. Action: canonicalise non-value-adding facets, noindex sort/order, allow only high-value facets (brand, top-level category) to remain indexable.
1,820 out-of-stock products still indexed
Out-of-stock products without OutOfStock schema or 301 redirect: 1,820 zombie pages dragging quality signal. Action: emit Product schema with availability=OutOfStock for temporary OOS; 301 to parent category for permanently OOS.
340 products with zero internal links
Products live in DB and sitemap but not linked from any category, collection or related-product widget. Crawl budget wasted; ranking unlikely. Action: add to related-product carousels or hide from sitemap until linked.
Inconsistent variant handling — 412 products
Some products use offer-level pricing on one URL; others have variant URLs canonicalised to parent; others have variant URLs indexed independently. Inconsistency confuses Google. Pick one pattern site-wide.
The catalogue has 240 real category pages. The crawler found 38,420 crawlable URLs beneath them. That ratio — roughly 160 machine-generated URLs for every page a merchandiser actually designed — is what "crawl explosion" means in practice. It is not caused by one bad decision. It is caused by four independent filter dimensions being combinable, each of which is individually reasonable.
Colour, size, price band and sort order are each defensible as a shopper feature. The problem is combinatorial: a category with 8 colours, 6 sizes, 5 price bands and 4 sort orders produces 960 filter states before anyone has clicked a second page of results. Multiply that by 240 categories and the arithmetic writes itself. Every one of those states is reachable by a crawler if the filter is a plain link with a query string, and every one of them returns HTTP 200 with a slightly different subset of the same products.
The cost is not a penalty. Google has been explicit that there is no duplicate-content penalty and no punishment for having many URLs. The cost is arithmetic. Googlebot allocates a finite number of fetches to a host, governed by how fast the server responds and how much demand Google has for the site's content. If 38,420 of the available fetches go to filter permutations, they are not going to the 12,480 product pages, and they are certainly not going to the newly added ones. This is why only 72% of PDPs in this crawl are indexed while every category page is: the crawler never runs out of things to fetch, so it never gets far enough down the queue.
Crawl budget is a real constraint on catalogues of this size. It is largely irrelevant on a 200-page brochure site, and anyone applying the advice in this section to a small site is optimising something that is not broken. The threshold at which it starts to matter is roughly where a site has tens of thousands of URLs, or a smaller number of URLs that change frequently — which is exactly the profile of a live ecommerce catalogue with fluctuating stock and prices.
The instinct is to block everything with a query string. That is usually wrong, because some facet combinations correspond to real, high-volume search demand and deserve to be landing pages in their own right. The test is not technical, it is commercial: does anybody search for this combination?
Facets with independent search demand
Brand within a category ("Nike running shoes"), and top-level category filters, typically have their own keyword volume, their own competitors in the SERP, and a coherent set of products. These should be crawlable, indexable, self-canonicalising, and ideally rendered as clean static paths rather than query strings — /running-shoes/nike/ rather than /running-shoes?brand=nike.
Facets that are genuine subsets but have no demand
A colour-plus-size-plus-price combination is a real filtered view a shopper wants, but nobody searches for it. Leave it crawlable so a shopper's shared link still works and still resolves, but point the canonical at the parent category. Accept that Google may sometimes disagree with the canonical — it is a hint, not an instruction — and rely on the next tier for the URLs where you cannot tolerate that.
Parameters that change presentation, not content
sort, order, view, page-size and session identifiers produce a re-ordered version of an identical product set. They have no search demand, no canonical ambiguity worth resolving, and they multiply faster than anything else. These are the correct target for a robots.txt Disallow, because the goal is to stop the fetch happening at all. The directives suggested above do precisely this.
One caveat that catches teams out: a URL blocked in robots.txt can still appear in the index as a bare URL with no snippet, if enough pages link to it. If a URL must be removed from the index rather than merely uncrawled, it has to stay crawlable and carry a noindex tag — because Google cannot read a noindex on a page it has been forbidden to fetch. Blocking and noindexing the same URL is a contradiction, and the block wins.
"Out of stock" is not one state. Treating it as one state is why this crawl found 1,820 pages that tell a shopper the item cannot be bought while telling Google, in their structured data, that it is available. The visible page and the schema disagree, which is the single most damaging thing a Product page can do — it is the condition that gets a merchant's items disapproved in Google's shopping surfaces.
| Situation | HTTP | Indexable? | Schema availability | Sitemap |
|---|---|---|---|---|
| Temporarily out of stock, returning | 200 | Yes | OutOfStock | Keep |
| Discontinued, direct successor exists | 301 to successor | n/a | n/a | Remove |
| Discontinued, no successor | 301 to category, or 410 | n/a | n/a | Remove |
| Seasonal, returns annually | 200 | Yes | OutOfStock | Keep |
The temptation is to 404 everything that cannot be bought today. Resist it for seasonal and restocking lines: a page that has accumulated links, reviews and ranking history for two years is worth keeping alive through a six-week stockout, and a returning 404 throws that away and forces you to earn it again. Keep the URL, keep the content, mark it OutOfStock honestly, and surface alternatives on the page so the visit is not wasted.
The opposite mistake is hoarding. A catalogue that never retires anything accumulates thousands of pages for products that will never be sold again, each one competing for crawl allocation against products that are actually in stock. If a line is dead, redirect it to its successor in a single hop. Redirecting does not cost you anything: Google has confirmed that no PageRank is lost through a 301 or a 302. What redirects do cost is latency and fragility, so redirect to the final destination directly rather than adding a hop to an existing chain, and update the internal links that point at the old URL so the redirect is a safety net rather than the primary path.
An orphan is a URL that exists in the database and the XML sitemap but is not linked from anywhere on the site. The 340 found here are typically the residue of category restructures, products whose category was deleted, discontinued-then-reinstated lines, and items whose only route in was a homepage promotion that ended.
A sitemap makes a URL discoverable. It does not make it important, and it gives Google nothing to work with beyond the URL string and whatever is on the page. Internal links do three things a sitemap cannot: they pass ranking signal from the pages that already have it, they describe the destination through anchor text and surrounding copy, and they place the product in a hierarchy so that its topical context is unambiguous. A product linked from "Men's waterproof jackets" is understood in a way that a product reachable only from an XML file is not.
Anchor text is worth a sentence of correction here, because a persistent myth says generic anchors like "click here" waste link equity. They do not change how much value is passed. What anchor text does is describe the destination — to search engines and, far more consequentially, to screen-reader users who navigate by pulling up a list of links on the page. "Waterproof jacket, navy, £89" is better than "view product" for both audiences, and the accessibility case is the stronger one.
The remedy for an orphan is architectural, not cosmetic. Adding it to a related-products carousel that renders client-side after a user interaction does not create a crawlable link. The product needs to appear in the served HTML of a category listing, a collection, a hub page or a related-product module — somewhere a crawler encounters it without executing an interaction. If a product genuinely belongs nowhere, that is a merchandising decision that should end in a redirect, not in a sitemap entry.
A 94% validation pass rate means 94% of PDPs emit Product JSON-LD that parses and carries the fields Google requires for a product result. It does not mean 94% are eligible for rich results, and it does not mean the data is correct. Validation checks structure. It cannot check whether the price in the markup matches the price on the page.
| Property | Status | Why it matters |
|---|---|---|
| name | Required | Must match the visible product title on the page. |
| image | Required | High-resolution, crawlable, not blocked in robots.txt. |
| offers.price / priceCurrency | Required | Must equal the price a shopper sees. A stale cached value is worse than no markup. |
| offers.availability | Critical | InStock / OutOfStock / PreOrder. The 1,820 mismatches above are all failures of this one field. |
| sku / gtin | Strongly advised | Lets Google match the item to the same product sold elsewhere. Missing GTINs are the most common cause of poor product matching. |
| brand | Strongly advised | Entity disambiguation. Empty strings are worse than omission. |
| aggregateRating / review | Optional | Only where genuine reviews are visible on the page. Marking up ratings a user cannot see is a policy violation. |
The rule that governs all of these: structured data must describe what is on the page. It is a machine-readable restatement of visible content, not an additional channel for claims the page does not make. This matters for AI answer engines too, though not for the reason usually given. There is no verified evidence that assistants preferentially cite pages carrying dense structured data. What is true and defensible is narrower: retrieval systems generally parse served HTML rather than rendering a page like a browser, so JSON-LD present in the initial response is available to them, and it disambiguates — it says unambiguously that £24.99 is a price and not a fragment of prose. That is a clarity benefit, not a volume game, and adding schema types the page does not support gains nothing.
These five issues are not independent, and fixing them in the wrong order wastes weeks. Faceted crawl suppression comes first, because until Googlebot stops burning its allocation on filter permutations, nothing else you fix will be recrawled promptly enough to measure. Expect the recrawl of the remaining PDPs to lag the robots.txt change by days to weeks on a catalogue this size.
Suppress the sort/order/view parameters in robots.txt
Highest volume, zero risk, immediate effect on fetch allocation. Verify in Search Console's crawl stats that fetches to those patterns fall to zero before moving on.
Fix the 1,820 availability mismatches
This is a correctness bug with commercial consequences in shopping surfaces, not an optimisation. It is also usually a single template change plus a cache-invalidation fix, so the effort-to-impact ratio is the best on the list.
Standardise variant handling across the 412 inconsistent products
Pick one of the three patterns and apply it everywhere. The inconsistency is the defect; the specific choice matters less than making it once.
Link the 340 orphans, or retire them
Cheap, mechanical, and it converts dead inventory into pages with a chance of ranking.
Promote the high-demand facets to static category paths
The slowest and most valuable piece. Brand-within-category pages built as real, self-canonicalising URLs with unique copy are the only part of this list that creates new ranking surface rather than protecting what exists.
Does having 38,000 faceted URLs cause a Google penalty?
No. There is no penalty for faceted URLs, and there is no duplicate-content penalty in Google Search. What happens instead is mundane and still expensive: Googlebot spends its fetch allocation on near-identical filter combinations rather than on the product detail pages that can actually rank, and consolidation signals get spread across many similar URLs so Google has to guess which variant is canonical. The cost is slow discovery of new and updated products, stale prices in the index, and unpredictable canonical selection. You fix it by reducing how many facet URLs are crawlable and indexable, not by fearing a punishment that does not exist.
Should I use robots.txt or noindex to control faceted URLs?
They do different jobs and they conflict if you use them together. A robots.txt Disallow stops the crawl, which is what you want when the URLs are worthless and numerous, such as sort, order and view parameters. But a disallowed URL can still be indexed without content if other pages link to it, and Google can never see a noindex tag on a page it is not allowed to fetch. A noindex meta tag requires the URL to remain crawlable, so it suits a smaller set of URLs you want fully dropped from the index. The practical split used in this report is: robots.txt Disallow for the high-volume, zero-value parameters; canonical tags for facet combinations that are genuine variants of a category; and noindex only where a crawlable URL must be kept out of the index for a specific reason.
Is rel=canonical enough on its own to fix faceted navigation?
Not on a catalogue this size. A canonical is a hint, not a directive: Google may choose a different canonical if the signals disagree, for example when the supposed duplicate has its own internal links, its own title and its own set of products. More importantly, a canonical does nothing for crawl cost, because Google must fetch the URL to read the canonical in the first place. Canonicals are the right tool for tidying up a few hundred legitimate near-duplicates. They are the wrong tool for suppressing tens of thousands of machine-generated combinations, which should not be reachable by a crawler at all.
What is the correct way to handle an out-of-stock product page?
Split it by intent. If the product is temporarily out of stock and will return, keep the URL live, return HTTP 200, keep it indexable, and set the Product offer availability to https://schema.org/OutOfStock so the structured data matches the visible page. If the product is permanently gone and has a direct successor, 301 redirect to that successor. If it is permanently gone with no successor, 301 to the most specific relevant category, or return 410 Gone and remove it from the sitemap. The failure mode this report flags is the 1,820 pages doing none of these: they still say InStock in the markup while the page tells a shopper the item cannot be bought.
Do 301 redirects for dead products lose link equity?
No. Google has stated that no PageRank is lost through a 301 or a 302 redirect. The difference between them is which URL gets indexed, not how much value is passed. Redirects still matter for other reasons: long chains add latency for users, waste fetches, and Google will stop following a chain after roughly ten hops, so a chain that grows over years of catalogue churn eventually breaks silently. Redirect discontinued products directly to their final destination in one hop and update internal links to point at the target rather than the redirect.
Why do orphan products matter if they are in the XML sitemap?
A sitemap tells Google a URL exists. Internal links tell Google how important it is and give it context from the anchor text and the surrounding page. The 340 orphan products in this crawl are discoverable but contextless: nothing on the site says which category they belong to, which products they compete with, or why anyone should reach them. They are also invisible to shoppers, which means they earn no engagement, no reviews and no links. Sitemap inclusion is a discovery mechanism, not a substitute for architecture. Put orphan products into a category listing or a related-product module, or accept that they will not perform and remove them.
Which variant architecture should an ecommerce site use?
Any of the three common patterns can work; mixing them cannot. Pattern one: a single product URL with every variant expressed as a separate offer in the Product schema, with variants selected by JavaScript or by a parameter that is canonicalised to the parent. Pattern two: one indexable URL per variant, each with its own Product schema, its own title and its own canonical pointing at itself, used when variants have real search demand, such as colour-specific fashion lines. Pattern three: parent page indexable, variant URLs canonicalised to the parent. This crawl found 412 products using different patterns from each other, which is the actual defect: Google is being given contradictory instructions about which URL represents the product.
Related Demo Reports
Run Ecommerce Crawl on Your Own Site
Get your real ecommerce crawl with PDP coverage, schema validation, faceted nav explosion analysis and orphan-product detection.