How to Fix Magento / Large-Catalogue Crawl Issues
Magento catalogues with thousands of products combined with layered (faceted) navigation produce millions of crawlable URLs Google never wants. Crawl budget gets consumed on filter combinations rather than canonical product pages. This guide covers Magento-specific crawl fixes. Pair with ecommerce crawl guide.
Step-by-step: How to fix Magento large-catalogue crawl
- Audit current crawl situation. Search Console → Settings → Crawl stats. Note: total URLs crawled per day, % of crawls returning 200 vs 3xx/4xx, % of crawl budget on /catalogsearch/, /catalog/category/view/, filter URLs. Magento sites commonly burn 40-80% of crawl budget on non-canonical filter URLs.
- Identify faceted URL patterns. Layered navigation generates URLs like ?cat=23&color=red&size=large&price=50-100. Each combination is a crawlable URL. Catalogue with 5 filters × 4 options each = 1024 combinations per category. Multiply by category count: easily millions of URLs.
- Configure robots.txt. Block crawler from non-canonical filter patterns: 'Disallow: /*?p=', 'Disallow: /*?cat=', 'Disallow: /*?color=', 'Disallow: /catalogsearch/', 'Disallow: /customer/'. Allow only canonical category and product URLs.
- Set canonical URLs on faceted pages. Magento layered navigation should output rel=canonical pointing to the parent category. Verify in HTML source. If missing, use Magento SEO extensions (Mageworx SEO Suite, Mirasvit SEO Suite) or custom module. Without canonicals, even blocked URLs can fragment ranking signals.
- Configure XML sitemap. Magento generates sitemap via admin → catalog → google sitemap. Configure: products in chunks of 50,000 (Google's limit per file). Include only canonical product and category URLs. Exclude filter URLs entirely. Submit via Search Console.
- Disable layered nav for crawlers. Some Magento sites detect crawler user-agents and serve simpler navigation (no filter links). Reduces crawl explosion. Verify cloaking concerns: if implementation differs significantly between user and crawler view, Google may flag. Conservative: use robots.txt + canonical + nofollow on filter links instead.
- Monitor. Search Console crawl stats weekly. Index Coverage report → identify newly indexed filter URLs (should be zero) and dropping non-canonical URLs (good signal). Goal: 90%+ of crawl budget on canonical product and category URLs.
- Address performance. Magento can be slow under crawl load. Cache configuration (Varnish, Redis), database tuning, CDN for static assets. Slow crawl response triggers Google to reduce crawl rate, which reduces freshness on legitimate product pages.
🏢 Audit Magento crawl
Find crawl-budget waste and faceted URL issues in your Magento store.
Run Magento Crawl Audit →Frequently Asked Questions
Why does Magento generate so many crawlable filter URLs?
Layered navigation builds query-string URLs for each filter combination. Default Magento behaviour treats every combination as a unique URL. Without crawler controls, Google attempts to crawl every combination. Catalogues with 20,000 products × 5 filters easily produce 10+ million crawlable URLs.
Should I block faceted URLs in robots.txt or set them noindex?
Both, in layers. robots.txt blocks crawl (saves crawl budget). Noindex via meta tag handles URLs already indexed (signals removal). For Magento at scale: robots.txt for crawl prevention + canonical to parent category for ranking signal consolidation. Noindex specifically on URLs Google has already indexed and you want removed.
Magento 1 vs Magento 2 crawl differences?
Magento 2 has better SEO defaults (rel=canonical on category filter pages, structured data extensions). Magento 1 (deprecated, end-of-life June 2020) often requires more manual SEO configuration. Migrating to Magento 2 (or Adobe Commerce) recommended both for SEO and security. Some sites on Magento 1 should plan migration urgently.
Best Magento SEO extensions for crawl management?
Mageworx SEO Suite — comprehensive (canonicals, faceted SEO, structured data, sitemap). Mirasvit SEO Suite — similar coverage. Amasty SEO Toolkit — meta-tag focused with crawl features. Most large Magento stores need one of these; native Magento SEO isn't sufficient at scale.
How long does Magento crawl-budget recovery take?
30-90 days. Google takes time to re-evaluate crawl rate. After robots.txt blocking + canonical updates, observe: Search Console crawl stats showing reduced filter-URL crawling, Index Coverage showing decreasing non-canonical URLs, organic traffic stabilising or improving as canonical pages get more frequent crawls.