Fixing Magento and Large Catalogue Crawl Issues
Magento Open Source and Adobe Commerce are built for big catalogues, and they are just as good at producing big numbers of URLs. Layered navigation turns every filterable attribute into a parameter, sort and page-size options multiply each category, category paths can give one product several addresses, and a multi-store setup repeats all of it per website.
On a catalogue of tens of thousands of products this can leave crawlers working through millions of combinations. This guide walks through the admin settings, attribute choices, robots.txt rules and sitemap configuration that bring a Magento crawl under control, with notes for multi-store installs.
Where Magento's extra URLs come from
A typical Magento 2 storefront exposes these URL families. Names are defaults; extensions and custom themes may add or rename parameters.
| Family | Example | Cause |
|---|---|---|
| Layered navigation | /footwear.html?color=49&size=167 | Each filterable attribute becomes a parameter, with option IDs as values |
| Price filter | ?price=50-100 | Price ranges in layered navigation |
| Category filter | ?cat=23 | Anchor categories showing child categories as a filter |
| Toolbar options | ?product_list_order=price&product_list_dir=desc&product_list_limit=36&product_list_mode=list | Sort, direction, page size and grid or list view |
| Pagination | ?p=5 | Category pages |
| Product via category path | /footwear/boots/ridgeway-boot.html | "Use Categories Path for Product URLs" and category/product rewrites |
| System URLs | /catalog/product/view/id/412 | Links built without a URL rewrite, often from extensions or imports |
| Search | /catalogsearch/result/?q=boots | Site search, plus advanced search and popular terms pages |
| Store switching | ?___store=fr&___from_store=default | Store view switcher links |
Toolbar parameters are the quiet multiplier. Four sort orders, two directions, three page sizes and two view modes give 48 variants of every category page before any filter is applied, and each paginated page has the same set.
The catalogue SEO settings that matter
Most of Magento's own URL behaviour is set in the admin under Stores, Settings, Configuration, Catalog, Catalog, Search Engine Optimization. The key options:
- Use Categories Path for Product URLs. Set to No so product links use the
short
/product-key.htmlform. With Yes, the product link depends on the category the shopper came through, creating duplicates. - Use Canonical Link Meta Tag For Products. Set to Yes. Product pages then declare the short URL as canonical even if reached another way.
- Use Canonical Link Meta Tag For Categories. Set to Yes. Category pages with layered navigation or toolbar parameters then point at the clean category URL.
- Create Permanent Redirect for URLs if URL Key Changed. Keep this on so editing a URL key leaves a 301 behind rather than a 404.
- Generate "category/product" URL Rewrites. Available in recent 2.x releases. Setting
it to No stops Magento creating category-path rewrites for every product, which also shrinks
the
url_rewritetable on large catalogues. Test on staging first, because existing category-path URLs that have links will need redirects.
Settings can be overridden at website or store view scope, so check the scope switcher at the top of the configuration page for each website you run.
Layered navigation: fewer filterable attributes
Every attribute set to appear in layered navigation adds a dimension of URLs. Many catalogues have attributes marked filterable that shoppers rarely use, often left over from an import where every attribute was switched on.
For each attribute, go to Stores, Attributes, Product, open the attribute and look at Storefront Properties. The option Use in Layered Navigation can be Filterable (with results), Filterable (no results) or No. Filterable (no results) shows options that return nothing, which creates empty pages, so prefer with results where you keep filtering. Use in Search Results Layered Navigation controls the same for search result pages.
Then decide what happens to the filtered URLs that remain:
- With category canonicals on, filtered pages already point at the clean category. That consolidates signals but they are still crawled.
- For a handful of combinations that deserve to rank, such as a brand within a category, create proper categories or landing pages with their own URL keys and content. Many SEO extensions offer "SEO-friendly layered navigation" that rewrites chosen filters into paths; use that selectively, not for every attribute.
- For attributes that should never be crawled, add robots.txt rules for their parameter names once any indexed versions have dropped out.
The general reasoning for choosing which combinations to keep is in Fixing faceted URL explosion.
robots.txt in Magento
Magento manages robots.txt in the admin under Content, Design, Configuration. Edit
the row for the website or store view you need, open Search Engine Robots, and use Edit
custom instruction of robots.txt File. The same section holds Default Robots,
which sets the meta robots value for pages that do not set their own; make sure the production website
is INDEX, FOLLOW and not a value copied from staging. A physical robots.txt file
in the web root, if present, may be served instead of the admin version, depending on your server setup,
so check which one is live by requesting it in a browser.
A starting point to adapt:
User-agent: *
Disallow: /catalogsearch/
Disallow: /search/
Disallow: /checkout/
Disallow: /customer/
Disallow: /wishlist/
Disallow: /sendfriend/
Disallow: /*product_list_order=
Disallow: /*product_list_dir=
Disallow: /*product_list_limit=
Disallow: /*product_list_mode=
Disallow: /*___store=
Disallow: /*SID=
Sitemap: https://www.example.co.uk/sitemap.xml- It does not block layered navigation attributes or
?p=. Add attribute parameters individually when you have decided, and keep pagination crawlable. - Never block
/static/,/media/or/pub/; crawlers need the CSS, JavaScript and images to render pages. - Blocked URLs can still be indexed from links without content. For patterns already indexed, let canonicals or noindex work first.
Multi-store and multi-website installs
One Magento install often runs several websites, store views and languages. Each is its own crawl surface.
- robots.txt per website. The Search Engine Robots setting can differ per website scope. If all websites share one domain with store codes in the path, they share one robots.txt, so rules must work for all of them.
- Store codes in URLs. The option Add Store Code to Urls under Stores, Configuration, General, Web, Url Options changes every URL on the site. Do not toggle it on a live store without a full redirect plan.
- Store switcher links carry
___storeparameters that crawlers can follow endlessly between views. Block them in robots.txt or make the switcher link directly to the equivalent page in the other store view. - Language versions need hreflang annotations, which Magento does not output by default; an extension or custom module is usual. See the hreflang guides for detail.
- One sitemap per store view, each listing only that view's URLs and referenced from the right robots.txt.
Sitemaps for large catalogues
Magento generates XML sitemaps under Marketing, SEO and Search, Site Map, with options in Stores, Configuration, Catalog, XML Sitemap. There you can set which content types are included, split limits for large files, and Enable Submission to Robots.txt so the sitemap line is added automatically. Scheduled generation depends on Magento cron running; if cron is broken, sitemaps quietly go stale and keep listing products you disabled months ago.
Check that sitemaps list only canonical, indexable URLs: short product URLs, clean category URLs, no disabled products, no category-path duplicates. A sitemap full of URLs that redirect or return 404 makes it harder to see real problems in Search Console. The sitemap size guide covers splitting very large sitemaps.
The fix, step by step
Step 1
Measure the crawl by URL family
Run
the Ecommerce Crawl to list the URL families your store exposes, then pull a month of server logs and
count requests from verified Googlebot by pattern: layered navigation parameters, toolbar parameters,
?p=, category-path product URLs, /catalog/product/view/, search and ___store.
Compare the total with your real count of products and categories. This baseline tells you which families
to tackle first and lets you prove the improvement to others later.
Step 2
Set the catalogue SEO options
In Stores, Configuration, Catalog, Catalog, Search Engine Optimization, set Use Categories Path for Product URLs to No, turn on canonical tags for products and categories, and keep permanent redirects on URL key changes. Consider turning off category/product URL rewrites after testing. Check every website and store view scope, then view source on a product, a filtered category and a paginated category to confirm the canonicals point where you expect.
Step 3
Cut back filterable attributes
List every attribute set to appear in layered navigation and remove those shoppers rarely use, changing Use in Layered Navigation to No. Switch any Filterable (no results) attributes to Filterable (with results) so empty option pages are not offered. For the few filter combinations with real search demand, create proper categories or landing pages instead of relying on parameter URLs, and give each its own introduction and meta title.
Step 4
Handle sort, limit, mode and search URLs
Confirm toolbar URLs with product_list_order, product_list_dir,
product_list_limit and product_list_mode carry the clean category canonical.
Consider reducing the page size options so fewer variants exist at all. Make sure search result pages
are not linked in bulk from templates, disable the popular search terms page if you do not use it, and
prepare robots.txt rules for these patterns ready for the next step.
Step 5
Write robots.txt per website scope
In Content, Design, Configuration, open each website's Search Engine Robots section, confirm Default Robots is INDEX, FOLLOW on production, and add rules for search, customer and checkout paths, toolbar parameters and store switcher parameters. Leave pagination and static assets open. Request the live robots.txt in a browser to confirm the admin version is served, then check it in the Search Console robots.txt report for each domain.
Step 6
Generate clean sitemaps on a schedule
Configure one sitemap per store view under Marketing, SEO and Search, Site Map, and set generation to run on a schedule in the XML Sitemap configuration. Confirm Magento cron is running so sitemaps actually refresh. Open each generated sitemap and spot-check that it lists only short product URLs and clean categories, with no disabled products or category-path duplicates, then submit each one in Search Console under the matching property.
Step 7
Test on staging and monitor after release
Apply configuration and attribute changes on a staging copy first, reindex, and crawl it to catch broken links or missing redirects before production. After release, flush caches, watch Search Console Crawl stats and page indexing weekly, and compare fresh log counts with your baseline. Recheck after every extension install or upgrade, because SEO, filter and multi-language extensions often change canonical and URL behaviour without warning.
1. Should Use Categories Path for Product URLs be on or off in Magento?
For most stores, off. With it switched off, every product link uses a short URL such as /ridgeway-boot.html, which stays the same whichever category the shopper browsed through. With it on, the product can appear at several category-path URLs, and the canonical tag has to do the work of consolidating them. If your store has run with it on for years and those URLs have links, plan redirects before switching rather than simply turning it off.
2. Does Magento add canonical tags to layered navigation pages?
Yes, when Use Canonical Link Meta Tag For Categories is enabled, filtered and sorted category pages declare the clean category URL as canonical. That consolidates signals, but the filtered URLs can still be crawled, so on very large catalogues you may also need to reduce filterable attributes and add robots.txt rules. Check the source of a filtered page after any extension install, because SEO and navigation extensions sometimes change this behaviour.
3. Where is robots.txt edited in Magento 2?
In the admin under Content, Design, Configuration. Choose the website or store view row, edit it, and open the Search Engine Robots section, where you will find Default Robots and Edit custom instruction of robots.txt File. Depending on the server configuration, a physical robots.txt file in the web root may be served instead, so always request /robots.txt in a browser after saving to confirm which version is live, then check it in Search Console.
4. Why are disabled products still in my Magento sitemap?
Usually because the sitemap is not being regenerated. Scheduled sitemap generation relies on Magento cron, and if cron has stopped or the schedule was never configured, the files on the server keep their old contents indefinitely. Check the generation time shown under Marketing, SEO and Search, Site Map, make sure cron jobs are running, and generate the sitemaps again. Then confirm disabled and out-of-scope products no longer appear in the output.
5. How many filterable attributes should a Magento category have?
There is no fixed number. Keep filters that shoppers actually use to narrow a category, such as size, colour, brand or price for clothing, and remove attributes that exist for internal reasons or were switched on in bulk during an import. Your site search and analytics can show which filters are used. Every filterable attribute adds another dimension of crawlable URLs, so each one should earn its place for shoppers.
6. Is crawl budget a real concern for Magento stores?
For larger stores, often yes. Google says crawl budget mostly matters for large sites or sites whose pages change often, and Magento stores are frequently both. Tens of thousands of products combined with layered navigation, toolbar options and several store views can produce far more URLs than crawlers will fetch regularly. If new products take a long time to be indexed, or logs show most Googlebot requests going to parameter URLs, the fixes on this page are worth doing.
7. Can I change Magento URL settings on a live store safely?
Some settings are low risk, such as enabling canonical tags. Others change URLs across the whole store, including category paths for product URLs, category/product rewrites and store codes in URLs. For those, test on a staging copy, map old URLs that have traffic or links to their new addresses, add redirects, reindex and clear caches. Make the change outside peak trading, then watch Search Console and server logs closely for several weeks.
The Ecommerce Crawl maps the URLs on your store, spots faceted navigation getting out of hand, finds duplicate product URLs and measures how efficiently your crawl is spent.
Open Ecommerce Crawl Fixes →