Fixing Faceted URL Explosion
Faceted navigation lets shoppers narrow a category by colour, size, brand, price and more. Each choice usually adds a parameter to the URL, and because choices can be combined, reordered and repeated, a few filters can produce more addresses than a crawler could ever finish.
This guide deals with the crawl side of that problem: measuring how many filter URLs your shop exposes, deciding which few deserve to be crawled, and stopping the rest with precise robots.txt rules and template changes, so crawlers spend their time on products instead of permutations.
- What this guide covers, and what it does not
- Why a few filters become thousands of URLs
- Recognising the explosion in your data
- The controls, ranked by how much crawl they save
- Writing robots.txt rules that match exactly what you mean
- Empty combinations and pagination inside filters
- The fix, step by step
- Platform notes for the crawl side
- Questions and answers
What this guide covers, and what it does not
Filters cause two distinct problems. One is on the page: whether filters are useful to shoppers and whether a filtered view deserves its own title, heading and text. That side is covered in Fixing faceted navigation. The other is the crawl: how many URLs the filters generate and how much crawler attention they absorb. This guide is only about the second. The two meet in one list, the filtered views you actually want in search results, and that list is the input to everything below.
Why a few filters become thousands of URLs
The growth is multiplicative, not additive. Suppose a Boots category has five filters: colour (10 values), size (12), brand (15), material (6) and heel height (4). If a shopper can pick one value from each filter or leave it unset, the number of distinct filtered views is:
(10+1) x (12+1) x (15+1) x (6+1) x (4+1) = 80,080That is for one category, before anything else multiplies it. Real shops add further multipliers:
- Multi-select. If shoppers can pick several colours at once, each filter contributes every subset of its values rather than one value, and the count grows far beyond the figure above.
- Order. If
?colour=black&size=7and?size=7&colour=blackboth work, every combination exists several times over. - Sort and view. Five sort orders and two grid layouts multiply the total by ten.
- Pagination. Every filtered view that spans several pages adds a
?page=sequence. - Empty results. Most combinations match no products at all, yet many platforms still return a 200 page for them.
Search engines cannot tell in advance which of these URLs are worth fetching. They learn by crawling samples, and on a large shop that sampling alone can take a noticeable share of the crawl. Google's own documentation on faceted navigation warns that letting these URLs be crawled increases load on your server and can slow the discovery of new pages.
Recognising the explosion in your data
You do not need server logs to see the problem, although they help. Look for these signs:
| Where | Sign |
|---|---|
| Ecommerce Crawl | Far more URLs found than products and categories combined, with most sharing a category path and differing only in query string. |
| Crawl Stats (Settings in Search Console) | A high and steady discovery share, and example URLs full of filter parameters. |
| Page indexing report | Large counts under "Duplicate without user-selected canonical", "Alternate page with proper canonical tag" or "Discovered – currently not indexed", with filtered URLs in the examples. |
| Server logs | Googlebot requests concentrated on parameter URLs rather than product pages. The crawl budget guide shows how to group log lines into families. |
A useful quick test: open a category, pick three filters, then change their order in the address bar by hand. If the page still loads with the same products and a 200 status, your shop accepts every ordering and the count above applies in full.
The controls, ranked by how much crawl they save
Each control has a different effect on crawling. Google's faceted navigation documentation recommends
robots.txt disallow rules or URL fragments for filtered URLs that do not need to be indexed, and treats
canonical tags and rel="nofollow" on filter links as weaker, secondary measures.
- Do not create a crawlable link at all. Filters that are not on the allow-list can
be buttons or form controls that update results with JavaScript, or can store their state in a URL fragment
such as
#colour=black. Google generally ignores fragments when crawling, so the combinations never reach the queue. - Disallow the pattern in robots.txt. Compliant crawlers stop requesting matching URLs. This is fast and needs no template work, but blocked URLs can still be indexed without content if other sites link to them.
- Canonical to the unfiltered category. Consolidates signals and, over time, reduces how often the duplicates are recrawled, but every URL still has to be fetched at least once for the tag to be seen.
- Nofollow on filter links. Only helps if applied to every link to those URLs, and Google treats it as a hint.
Writing robots.txt rules that match exactly what you mean
Google and Bing support two wildcards in robots.txt: * matches any sequence of characters,
and $ anchors the end of the URL. When an Allow and a Disallow rule both match, Google uses
the more specific rule, meaning the longer one; if they are the same length, Allow wins. Paths and parameters
are case-sensitive.
The substring trap
A rule such as Disallow: /*size= matches any URL containing size= anywhere,
including ?pagesize=48 and a product handle like /products/oversize=fit. Anchor
parameters to the character before them instead. A parameter can follow either ? or &,
so each needs two lines:
User-agent: *
# sort and layout options: never worth crawling
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?view=
Disallow: /*&view=
# price slider values create an endless range
Disallow: /*?price=
Disallow: /*&price=
# any URL with two or more parameters
Disallow: /*?*&
# single-parameter views such as ?colour=black or ?page=2 stay crawlable
Sitemap: https://www.example.co.uk/sitemap.xmlRead that example as a pattern, not a template: your parameter names, and which single-filter views you keep, will differ. Notice that the rule blocking two or more parameters does the heavy lifting. Most explosion comes from combinations, so leaving single-filter views open and blocking every multi-parameter URL removes the bulk of the problem in one line.
Be careful with Allow lines. It is tempting to add Allow: /*?colour= to protect the colour
views, but that line is longer than /*?*&, so it would also win for ?colour=black&size=7&material=suede
and reopen every combination that starts with colour. Single-parameter URLs do not match the multi-parameter
rule in the first place, so they need no Allow line. If a two-filter view is on your allow-list, give
it a clean path of its own rather than trying to carve it out with wildcards.
Path-based filters
Some platforms and plugins put filters in the path, for example /boots/colour-black/size-7/.
The same logic applies, but the rules match path segments. If every filter segment starts with a known
prefix, Disallow: /*/size- blocks the size facet wherever it appears. Keep the prefixes
unique so they cannot match a real category name.
Empty combinations and pagination inside filters
Two details decide whether the allowed filter URLs stay under control.
Return 404 when a combination has no products
Google's guidance is to return a 404 status when a filter combination returns no results, rather than a 200 page saying "no products found" or a redirect to the category. A 200 page with nothing on it invites repeat crawling and may be reported as a soft 404. On a custom build this is a small change in the controller. On hosted platforms you may not be able to change the status code, which is another reason to stop linking to combinations that are not on the allow-list.
Keep filter pagination finite
Requesting page 500 of a filtered view that only has two pages should return a 404, not a copy of
the last page. Check this by hand: add &page=999 to an allowed filter URL. If you get
products back with a 200 status, crawlers can walk through as many page numbers as they find links to.
The fix, step by step
Step 1
Measure how many filter URLs are reachable
Run the Ecommerce Crawl and note how many URLs it finds compared with the number of products and categories you sell. Then open Crawl Stats in Search Console and look at example URLs under the discovery purpose. Finally, test a category by hand: reorder filter parameters in the address bar, add an impossible combination and request a very high page number. Record which of these return a 200 status, because each one multiplies the problem.
Step 2
Build a parameter inventory
List every query parameter or path segment your filters, sorts, views and pagination produce, one per row in a spreadsheet. For each, record its name, what it does, how many values it can take, whether shoppers can select more than one value, and which categories use it. Include parameters added by apps and plugins, such as currency switchers, wishlist tools or review widgets. This inventory is what your robots.txt rules and template changes will be built from.
Step 3
Write a short allow-list of crawlable filter URLs
Mark each filtered view you want search engines to reach, based on genuine search demand, such as a brand or colour within a category. Keep the list short: single-filter views are usually enough, and combinations of two filters should need a clear reason. Anything not on the allow-list is treated as shopper-only. Agree the list with whoever owns the page side, so allowed views get proper titles and text rather than template defaults.
Step 4
Make allowed URLs predictable
Ask
your developer to output allowed filter URLs in one fixed parameter order, using the standard &
separator, with lowercase values and no duplicate parameters. Any other ordering of the same filters
should redirect or canonicalise to the fixed form. Combinations with no matching products should return
a 404 status, and page numbers beyond the last real page should do the same. These changes
stop the allowed views from multiplying on their own.
Step 5
Stop templates linking to everything else
Change the filter template so only allowed views are rendered as ordinary <a href>
links. Other filter options can be buttons, checkboxes in a form, or links that keep their state after
a # fragment, so crawlers never see a new URL. Remove sort and layout options from crawlable
links entirely. This is the most effective single change, because a URL that is never linked is never
discovered in the first place.
Step 6
Block the remaining patterns in robots.txt
For patterns that are still linked, or that old links and external sites keep exposing,
add disallow rules. Anchor each parameter to ? and & so it cannot match
a longer parameter name by accident. Add a rule that blocks URLs with two or more parameters, and leave
single-parameter views unblocked instead of carving them out with broad allow rules. Remove any noindex
or canonical logic you were relying on for the same patterns, since blocked pages cannot pass it on.
Step 7
Test the rules and watch the crawl settle
Test every rule against a list of URLs that must stay crawlable and a list that must not, then publish the file. Check the robots.txt report in Search Console to confirm Google has fetched the new version. Over the next four to six weeks, the discovery share in Crawl Stats should fall and filtered examples should disappear from it. Re-run the Ecommerce Crawl to confirm the number of reachable URLs has dropped.
Platform notes for the crawl side
- Shopify ships a default robots.txt that already blocks some filter and sort patterns,
including combinations of several filters. Single filters are generally left crawlable. See Shopify
crawl issues for editing
robots.txt.liquidsafely. - WooCommerce filtering depends on the plugin, and some plugins create both parameter and path versions of the same filter. Inventory both before writing rules; the WooCommerce crawl guide covers attribute archives.
- Magento and Adobe Commerce layered navigation creates parameter URLs for every attribute marked filterable, across every store view. Reducing which attributes are filterable in the catalogue settings cuts the problem at source; the Magento guide has detail.
- Headless and custom builds have the most freedom. Keep filter state client-side by default and generate server-rendered links only for the allow-list.
1. How many filter URLs is too many?
There is no fixed number. The useful comparison is between crawlable URLs and the pages you actually want found: products, categories and a short list of filtered views. If the crawl finds ten or a hundred times more URLs than that, crawlers are spending most of their visits on combinations nobody needs. For a small shop this may never cause a visible problem, but on a large catalogue with daily stock changes it can delay new and updated products.
2. Should I use robots.txt or canonical tags for filter URLs?
They do different jobs, so choose per pattern. Robots.txt stops compliant crawlers fetching the URLs, which is what saves crawl, but it hides any tags on the page. A canonical tag lets Google fetch the page and consolidate it with the category, which helps with signals but still costs a crawl each time. For sort orders and multi-filter combinations with no search value, robots.txt is usually the better tool. For near-duplicates you want consolidated, use canonicals.
3. Will blocking filter URLs remove pages that are already indexed?
Not immediately, and sometimes not at all. A blocked URL that is already indexed can stay in results, often shown without a description, because Google can no longer fetch it to see that it has changed. If many filtered URLs are indexed and you want them gone first, leave them crawlable with a canonical or noindex until they drop out, then add the robots.txt rule. This takes longer but avoids stranded URLs.
4. Is it safe to block every URL with a question mark?
Rarely. Pagination, internal search, tracking-free campaign landing pages and some product variant URLs may all use parameters, and some platforms put essential pages behind query strings. A blanket rule such as Disallow: /*? can cut off legitimate pages and resources. Start from your parameter inventory, block named parameters and multi-parameter combinations, then check the views you need are still reachable, and test every rule against a list of URLs that must stay crawlable before you publish it.
5. Does rel nofollow on filter links stop crawling?
Only partly. Google treats nofollow as a hint and may still crawl the target, and the hint only helps if every link to those URLs carries it, including links in sitemaps, feeds and other sites. Google lists it as a secondary option in its faceted navigation guidance. Rendering non-allowed filters as buttons or fragment-based links is more reliable, because the crawler never sees a new URL to follow in the first place.
6. Why does Google still crawl filter URLs after I blocked them?
First check the rule really matches: parameter order, case and the character before the parameter all matter. Then check the robots.txt report in Search Console to confirm Google has the new file. Other crawlers that ignore robots.txt, or fake Googlebot user agents, can also show up in logs, so verify the IP addresses. Finally, Crawl Stats covers about 90 days, so older requests can remain visible for weeks after the change.
7. Do URL fragments really stop filter URLs being crawled?
Google says it generally does not support fragments in crawling and indexing, so a link to /boots#colour=black is treated as a link to /boots. That is why fragment-based filter state keeps combinations out of the crawl. The trade-off is that fragment URLs cannot rank as separate pages, so use fragments only for filters that are not on your allow-list, and keep real, server-rendered URLs for the filtered views you want in search results.
The Ecommerce Crawl maps the URLs on your store, finds faceted navigation explosion and duplicate product URLs, flags out-of-stock products, surfaces orphans and measures how efficiently your crawl is spent.
Open Ecommerce Crawl Fixes →