Ecommerce Crawl Fixes: Control the URLs Your Shop Creates
Shops generate URLs far faster than content sites. Every filter, sort order, variant and collection can create another address for what is really the same list of products, and search engines have to decide which of those addresses deserve their attention.
Our Ecommerce Crawl is built for stores: it maps the URLs on your shop, spots faceted navigation getting out of hand, finds duplicate product URLs, flags out-of-stock products, surfaces orphan products and measures how efficiently your crawl is spent. This page explains each finding and the tools you have to control it.
Where the extra URLs come from
A store with a few hundred products can easily expose tens of thousands of crawlable URLs without anyone deciding to create them. Almost all of them come from a handful of sources.
| Source | Example | Usually worth indexing? |
|---|---|---|
| Filter combinations | /boots?colour=black&size=7&material=leather | Occasionally, for combinations people really search for |
| Sort and view options | ?sort=price-asc, ?view=grid | No; same products in a different order |
| Product paths through collections | /collections/sale/products/boot | No; the plain product URL should be the one indexed |
| Variants | /products/boot?variant=123 | Sometimes, depending on how people search |
| Pagination | /boots?page=7 | Crawlable so products are found, rarely a landing page |
| Internal search results | /search?q=boot | No; Google advises against letting these be indexed |
| Tracking parameters | ?utm_source=newsletter | No |
The crawl report groups your URLs by patterns like these so you can see which family is growing out of control. The fix is almost never to delete pages. It is to decide, family by family, which URLs should be crawled, which should be crawled but consolidated, and which should not be crawled at all.
The four controls, and what each one actually does
Much crawl advice goes wrong because the controls are used interchangeably. They do different jobs, and some cancel each other out.
- robots.txt
Disallowstops compliant crawlers fetching matching URLs. It saves crawl effort, but a blocked URL can still appear in results if other pages link to it, and search engines cannot see any canonical or noindex tag on a page they are not allowed to fetch. - Canonical tags tell search engines which of several similar URLs you prefer. The duplicates still get crawled, but signals are consolidated onto the preferred version. Google treats the canonical as a strong hint, not an order.
- Meta robots
noindexkeeps a page out of the index but requires the page to be crawled so the tag is seen. It does not save crawl effort in the short term. - Not creating the URL at all, for example by applying filters without changing the address, or by using links that only exist for the filters you want indexed, is often the cleanest option, but it needs developer work.
Google retired the URL Parameters tool in Search Console in 2022, so parameter handling now has to be done on the site itself with the controls above. The Robots and Sitemap Guide covers the syntax.
The fix guides, by finding
URL growth and duplication
Fixing faceted URL explosion shows how to choose the handful of filter combinations worth indexing and stop the rest multiplying. Fixing duplicate product URLs covers one product reachable through several paths, pointing canonicals at the primary URL and, just as importantly, updating internal links to use it. Fixing variant URL strategy weighs separate URLs for sizes and colours against one page with selectable options. Fixing collection and category architecture deals with products that sit in several overlapping collections, such as Men's, Sale and Summer, and which collection should carry the main signals.
Product lifecycle
Fixing out-of-stock handling sets out the choices for products that are temporarily unavailable or gone for good. Fixing seasonal product handling covers keeping seasonal URLs alive between seasons so they keep their history. Fixing orphan product pages covers products listed in the sitemap that no category links to, often left behind when a collection is deleted.
Efficiency
Fixing crawl budget waste brings the rest together: reducing low-value URLs, making sure important pages are linked and in the sitemap, and checking the result in server logs and the Search Console Crawl Stats report.
By platform
Shopify covers collection-based
product paths, variant parameters, apps that add URLs and editing robots.txt.liquid. WooCommerce
covers overlapping product categories and tags, attribute archives and server rules for shaping the crawl.
Magento and large catalogues covers
layered navigation, multi-store setups and rules that work across very large catalogues.
Products that come and go
Content sites mostly add pages. Shops constantly add, pause and retire them, and each state change is a chance to lose rankings that took months to build.
- Temporarily out of stock. Keep the page live, say clearly when it will be back or
offer a back-in-stock alert, and set
availabilityin the Offer markup toOutOfStockorBackOrder. Removing the page only to recreate it next month throws away its history. - Discontinued with a direct replacement. A permanent (301) redirect to the successor product passes on most of the old page's value and helps anyone following an old link.
- Discontinued with no replacement. Redirect to the closest relevant category only if it genuinely serves the same need. Mass redirects of unrelated products to one category or the home page may be treated as soft 404s. Otherwise, a 404 or 410 with helpful links is honest and acceptable.
- Seasonal. Use a URL without a year, such as
/christmas-jumpers, keep it live all year with a note about when the range returns, and refresh the content each season.
Which findings to fix first
Crawl budget is mainly a concern for large sites. Google's own guidance says most sites with a few thousand URLs are crawled efficiently without special work. Shops become the exception quickly because of filters, so prioritise by scale.
- Important pages not being found. Orphan products and pages missing from the sitemap cost you directly, whatever the size of the shop.
- Duplicate product URLs in internal links. If your own templates link to collection paths rather than the canonical product URL, fix the template once and the duplication stops at source.
- Uncontrolled filter and sort URLs. The main cause of crawl waste on larger stores. Decide on rules per pattern and apply them.
- Out-of-stock and discontinued handling. Set a policy so staff handle every case the same way.
- Variant and collection structure. Worth getting right, but changes here affect many URLs at once, so plan them rather than rushing.
The Site Crawler fixes page covers general crawl problems such as broken links and redirect chains, which affect shops too.
Order of work for a crawl clean-up
- Run the Ecommerce Crawl and note the URL families with the most pages.
- Compare with the Search Console Crawl Stats and page indexing reports to see what Google is actually crawling and indexing.
- For each URL family, decide: index, consolidate with canonicals, block from crawling, or stop generating.
- Fix templates so internal links, sitemaps and canonicals all point to the same preferred URLs.
- Apply robots.txt rules last, after canonicals and links are right, and test them before publishing.
- Write down an out-of-stock and discontinued policy and share it with whoever manages products.
- Recrawl after a few weeks, then after any new app, theme change or range launch.
For background, the Site Crawler Guide, the Robots and Sitemap Guide and the Ecommerce SEO Guide go into more depth. Product markup for availability is covered in the Schema Builder Guide.
1. What is crawl budget and does my shop need to worry about it?
Crawl budget is the amount of crawling a search engine is willing and able to do on your site in a given period. Google says most small sites do not need to worry about it. Shops are different because filters, sorts and variants can multiply a few hundred products into many thousands of URLs, and crawlers may spend their time on those instead of your real pages. If your crawl shows URL counts far above your product count, it is worth attention.
2. Should I block filter URLs in robots.txt?
Often yes for filters and sorts that nobody searches for, because it stops crawlers wasting time on them. But blocking is a blunt tool. Blocked URLs can still appear in results if they are linked to, and search engines cannot see canonical or noindex tags on pages they may not fetch. Keep the few filter combinations with real search demand crawlable, block the rest by pattern, and test the rules before publishing them.
3. What should I do with a product that is out of stock?
If it is coming back, keep the page live, show the expected return date or a back-in-stock signup, and mark availability as out of stock in the Offer markup. If it has gone for good, redirect permanently to a close replacement where one exists. If nothing comparable exists, a helpful 404 or 410 page is acceptable. Avoid redirecting large numbers of unrelated products to the home page, which search engines may treat as soft 404s.
4. Why does the same product appear at several URLs on Shopify?
Shopify can show a product at its own address and also at addresses that include the collection it was reached through. By default the page carries a canonical tag pointing to the plain product URL, which helps. The remaining problem is usually the theme linking to the collection-based version, which sends crawlers to duplicates. Changing the theme so product links use the plain URL fixes this at source. The Shopify crawl guide shows where to look.
5. What is an orphan product and why does it matter?
An orphan product is a live product page that no other page on your site links to. It may still be listed in your sitemap, but without internal links search engines treat it as less important and customers browsing the site never find it. Orphans often appear when a collection is deleted or products are moved. The fix is either to link the product from a relevant category again or, if it is no longer sold, retire it properly.
6. How can I see what Googlebot is actually crawling?
Search Console's Crawl Stats report, found under Settings, shows crawl requests over time, response codes and file types, with examples of crawled URLs. For full detail, your server access logs record every request, and filtering them for verified Googlebot traffic shows exactly which URL patterns it spends time on. Comparing that with the Ecommerce Crawl results tells you whether your controls are working or whether crawlers are still stuck in filter URLs.
7. Should seasonal pages be deleted after the season?
Usually not. A page such as a Christmas range builds up links and ranking history over the years, and deleting it means starting again every autumn. Keep the page at a permanent URL without a year in it, leave it live with a short note about when the range returns and links to related products, then refresh the products and content before the season starts. That way the page is already trusted when demand rises again.
The Ecommerce Crawl maps the URLs on your store, finds faceted navigation explosion and duplicate product URLs, flags out-of-stock products, surfaces orphans and measures how efficiently your crawl is spent.
Run the Ecommerce Crawl →