Fixing Crawl Budget Waste on an Online Shop

Crawl budget is the number of URLs Googlebot can and wants to fetch from your site. Google is clear that most sites never need to think about it, so the first job is to check whether it is really your problem.

On a shop it often is, because filters, sorts, old product URLs and redirects can soak up requests that should go to live products. This guide shows how to confirm the problem with Search Console and your server logs, which kinds of waste to remove first, and how to prove the change worked.

By John Carlsson · Updated 28 September 2026 · About 15 minutes to read

📋 Demo 📘 Tutorial 📖 Guide 📚 Book 🔧 Fix

In this guide

Do you actually have a crawl budget problem?
How Google decides how much to crawl
Reading the Crawl Stats report
Measuring waste in your server logs
Where shop crawl goes to waste
Server speed, errors and conditional requests
The fix, step by step
Keeping crawl efficient after the clean-up
Questions and answers

Do you actually have a crawl budget problem?

Google publishes guidance on managing crawl budget and says plainly that it is aimed at large sites. If your pages are usually crawled on the day they are published, you do not need it. The guidance singles out three situations where it matters:

Very large sites, around a million unique pages or more, whose content changes roughly weekly.
Medium or larger sites, from about ten thousand unique pages, whose content changes daily, such as a shop with fast-moving stock and prices.
Sites where a large share of URLs sits in the Search Console status Discovered – currently not indexed.

The count that matters is not your number of products. It is the number of URLs a crawler can reach. A shop with 800 products and six filters can easily expose more than ten thousand crawlable addresses, which is why stores cross the threshold much sooner than blogs or brochure sites.

Symptoms that point elsewhere

Plenty of indexing problems are blamed on crawl budget when the cause is something else. Rule these out before spending days on logs.

What you see	Crawl budget?	More likely cause
Many pages in Crawled – currently not indexed	No. Google fetched them.	Google judged the pages not worth indexing: thin, near-duplicate or low-value content.
Many pages in Discovered – currently not indexed	Possibly	Crawl demand or capacity is limited, or the URLs look like low-value duplicates of pages already known.
New products take weeks to appear	Possibly	Weak internal linking to new arrivals, or products missing from the sitemap.
A small site with a few hundred URLs not being indexed	Almost never	Blocking rules, noindex tags, canonicals pointing elsewhere or quality issues.
Crawl requests dropped suddenly	Yes, capacity	Server errors, timeouts or a slower host making Googlebot back off.

💡 If Crawl Stats shows Googlebot fetching far more URLs per day than you have products and categories, you are almost certainly wasting crawl, even if indexing looks fine today.

How Google decides how much to crawl

Google describes crawl budget as the product of two things. Understanding which one limits you tells you which fixes will help.

Crawl capacity limit

This is how hard Googlebot can hit your server without hurting it. It rises when your pages respond quickly and consistently, and falls when responses slow down or the server returns errors. A shop on shared hosting that takes two seconds to generate each category page is telling Googlebot to go gently. Capacity problems show up as falling request counts, a rising average response time and host status warnings in Crawl Stats.

Crawl demand

This is how much Google wants to crawl. It depends on how popular your URLs are, how often they change and how many URLs Google believes exist. Demand is also where waste bites: if Google keeps discovering new filter, sort and session URLs, part of its interest in your site is spent confirming that those URLs are duplicates. Reducing the number of pointless URLs lets the same demand go to pages that earn money.

You cannot buy more budget or request it. You can make your server faster and more reliable, make each URL worth fetching, and stop creating URLs that are not.

Reading the Crawl Stats report

The report lives under Settings → Crawl stats in Search Console and is available for domain-level and top-level URL-prefix properties. It covers roughly the last 90 days. Four breakdowns matter for a shop:

By response. A healthy shop is dominated by 200 responses. A large share of 301, 302 or 404 responses means Googlebot is following old links, redirects or removed products.
By file type. HTML should be a big slice. If JSON, JavaScript or images dominate, check whether theme assets are uncacheable or change their URL on every deploy.
By purpose. Discovery means URLs Google had not crawled before; refresh means re-crawling known URLs. A large and constant discovery share on a site whose catalogue is not growing is a classic sign of URLs being generated from parameters.
Host status. Problems with robots.txt fetching, DNS or server connectivity appear here and directly reduce crawling.

Click into each row to see example URLs. These examples are a sample, not a full list, but they are usually enough to spot the pattern, such as dozens of ?sort_by= or ?page=38 URLs.

Measuring waste in your server logs

Crawl Stats shows trends and samples. Your access logs show every request, which is what you need to put a number on waste. Most hosts, including Plesk and cPanel servers, keep logs for each domain; ask your host or developer if you cannot find them. Take at least two weeks, ideally four.

Verify that it really is Googlebot

Anyone can send a Googlebot user agent. Genuine requests come from Google's published IP ranges and pass a two-way DNS check: a reverse lookup of the IP returns a host name ending in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name returns the same IP.

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Collapse URLs into families

Counting individual URLs is useless on a shop; you want patterns. This one-liner, run on a combined log format file, strips parameter values so that ?colour=red and ?colour=blue are counted together:

grep 'Googlebot' access.log \
  | awk '{print $7}' \
  | sed -E 's/=[^&]*/=/g; s#/products/[^/?]+#/products/*#' \
  | sort | uniq -c | sort -rn | head -40

Put the output into a spreadsheet with three columns: family, requests, and what the family is worth. You will usually find that a handful of families take most of the requests.

Family	Typical share on a messy shop	Should Googlebot fetch it?
Product pages, plain URL	Often surprisingly small	Yes, and more often
Category pages and their first few pages	Moderate	Yes
Filter and sort combinations	Frequently the largest	Only whitelisted combinations
Redirecting old URLs	Varies with site age	No; fix the links that lead there
404s for removed products	Varies	A few are normal; many mean links still point to them
Internal search, cart, account	Should be near zero	No

⚠️ Access logs contain IP addresses, which are personal data under UK GDPR. Filter to crawler traffic before sharing logs with an agency or contractor, and delete working copies when you are done. This is not legal advice.

Where shop crawl goes to waste

Once the logs are grouped, most waste falls into a few categories. Each has its own guide in this hub, so this section only sets out how to recognise them in the data.

Filter, sort and view parameters. Recognised by many URLs sharing a category path with different query strings. The rules for keeping the useful combinations and stopping the rest are in Fixing faceted URL explosion.
Redirect hops. Recognised by 301s on URLs your own templates still link to, such as old product handles or http:// links in product descriptions. Each hop costs a request before the real page is reached.
Dead products. Recognised by repeated 404s on product paths. If Googlebot keeps asking, something still links to them: an old sitemap, a feed, related-product widgets or blog posts.
Soft 404s. Empty category pages or discontinued products that return 200 with a "no products found" message. Google treats these as errors and may keep re-checking them.
Endless spaces. Calendars in booking widgets, infinite ?page= numbers that return content past the last real page, or session IDs added to links. These are covered under crawler traps.
Duplicate hosts. The shop answering on both www and bare domain, or http and https, without redirecting, which doubles every URL.

Server speed, errors and conditional requests

Capacity is the half of crawl budget that shop owners most often overlook. A few server-side changes can let Googlebot fetch more without any change to the URLs:

Cache category and product HTML where your platform allows it, so pages are not rebuilt from the database for every crawler request.
Return 503 during maintenance, with a Retry-After header, rather than a 200 page saying the shop is closed. Short 503 periods are fine; long ones cause pages to drop.
Do not use 403 or 404 to slow Googlebot down. Google's advice is that 500, 503 and 429 responses reduce crawling; other codes do not and can remove pages from the index. Google removed the crawl rate setting from Search Console in early 2024, so server responses are now the lever.
Support conditional requests. If your server honours If-Modified-Since or ETag headers and returns 304 Not Modified when a page has not changed, re-crawls cost you and Google less.
Keep asset URLs stable. Themes that append a new version string to every script and stylesheet on every deploy make Googlebot re-fetch them all.

The fix, step by step

Step 1

Check whether crawl budget is really the constraint

Open the Page indexing report in Search Console and compare the number of URLs under Discovered – currently not indexed with the number under Crawled – currently not indexed. A large discovered count on a site with many thousands of crawlable URLs points to a crawl problem. A large crawled count points to content quality instead. Then run the Ecommerce Crawl and compare the number of URLs it finds with the number of products and categories you actually sell.

Step 2

Pull three months of Crawl Stats

Go to Settings, then Crawl stats, and note total crawl requests, average response time and the split by response, file type and purpose. Screenshot each chart so you have a baseline to compare against later. Open the examples behind any row that looks large, such as 301 or discovery, and write down the URL patterns you see repeated. Check host status for robots.txt, DNS or connectivity problems before you go any further.

Step 3

Group verified Googlebot requests by URL family

Download at least two weeks of access logs, keep only requests from genuine Googlebot IP addresses after a reverse and forward DNS check, and strip parameter values so similar URLs collapse together. Count requests per family and sort them from largest to smallest. Label each family as worth crawling, worth crawling less, or not worth crawling at all. The top five families usually explain most of the waste on a shop.

Step 4

Cut the families that should not be crawled

For each family marked not worth crawling, choose one control: stop the templates generating links to it, block the pattern in robots.txt, or both. Do not add a noindex tag to URLs you also block, because Googlebot will never see it. Internal search, cart, checkout, account and sort URLs are safe to block. Filter combinations need a whitelist first, so work through the faceted explosion guide before blocking those patterns.

Step 5

Take redirects, errors and soft 404s out of internal links

Export the redirecting and 404 URLs from your log analysis and find where each is linked from, using the Ecommerce Crawl or a site crawler. Update menus, footers, product descriptions and related-product blocks to link straight to the final live URL. Make empty categories and permanently removed products return a real 404 or 410, or redirect them to a close replacement, instead of a 200 page with no products.

Step 6

Speed up responses and stop server errors

Check the average response time in Crawl Stats and look for any 5xx responses in your logs. Turn on page caching for category and product templates, move heavy apps or plugins off critical templates, and ask your host whether the server is running out of memory at busy times. Make sure planned maintenance returns 503 with a Retry-After header, and that your server returns 304 for unchanged pages when asked.

Step 7

Publish a clean sitemap and re-measure

Make sure your XML sitemap lists only canonical, indexable, live URLs that return 200, with accurate lastmod dates that change only when the page really changes. Resubmit it in Search Console. Wait four to six weeks, then repeat the log grouping and compare with your baseline. The share of requests going to plain product and category URLs should rise, and the wasted families should shrink.

Keeping crawl efficient after the clean-up

Crawl waste comes back whenever something new starts generating URLs. The usual culprits are a new filter app, a theme update that changes how links are built, a marketing tool that adds tracking parameters to internal links, or a seasonal range that is deleted instead of kept.

Re-run the log grouping monthly, or after any theme, app or platform change, and compare family sizes with last month.
Keep a short written list of URL patterns that should never be crawled and check new features against it before they go live.
Watch the discovery share in Crawl Stats. A jump without a matching jump in real products means a new URL source has appeared.
Keep internal links clean at source: templates should output the canonical URL for every product and category, with no tracking parameters.
Agree a policy for removed and out-of-stock products so staff do not create redirect chains or soft 404s one product at a time.

The Ecommerce Crawl guide explains how the crawl report groups URLs, and the Robots and Sitemap guide covers rule syntax and sitemap structure in more depth.

Questions and answers

1. Does a small shop need to worry about crawl budget?

Usually not. Google says most sites are crawled efficiently without any special work, and its crawl budget guidance is aimed at sites with tens of thousands of URLs or more. The catch for shops is that the relevant number is crawlable URLs, not products. A small shop with filters, sort options and tag pages can expose many thousands of addresses. If your crawl finds far more URLs than you have products and categories, it is worth checking.

2. What is the difference between Discovered and Crawled currently not indexed?

Discovered currently not indexed means Google knows the URL exists but has not fetched it yet, which can be a sign that crawling is limited or that the URL looks like a low-value duplicate. Crawled currently not indexed means Google fetched the page and chose not to index it, which is about the page itself, not crawl budget. The second status is usually fixed by improving or consolidating content, not by changing crawl rules.

3. Can I ask Google to crawl my shop more often?

Not directly. There is no setting that raises crawl budget, and Google removed the crawl rate setting from Search Console in early 2024. You influence crawling indirectly: a fast, reliable server raises the capacity limit, and useful, well-linked, frequently updated pages raise demand. Requesting indexing for individual URLs in the URL Inspection tool is fine for a handful of important pages, but it is not a way to process a whole catalogue.

4. Does blocking URLs in robots.txt free up crawl budget?

Yes, for Googlebot and other crawlers that follow the rules, blocked URLs are not fetched, so those requests can go elsewhere. The trade-off is that Google cannot see anything on a blocked page, including canonical and noindex tags, and blocked URLs can still be indexed without content if they are linked. Block patterns that have no search value, such as sort orders, internal search and cart pages, and use canonicals for near-duplicates you want consolidated.

5. Do noindex tags save crawl budget?

Not in the short term. Googlebot has to fetch a page to see its noindex tag, so noindexed URLs are still crawled. Over time Google tends to crawl long-standing noindexed pages less often, but it is not a reliable way to cut waste. If a URL pattern has no value at all, the stronger fix is to stop linking to it and, where appropriate, block it in robots.txt. Keep noindex for pages that people need but search results do not.

6. How do I find my server logs?

On most hosting control panels, such as Plesk or cPanel, each domain has an access log you can view or download from the logs section. Managed platforms such as Shopify do not give merchants raw server logs, so on those you rely on the Crawl Stats report and your crawl results instead. If a CDN such as Cloudflare sits in front of your site, its logs may be more complete than the origin server logs.

7. How long before I see the effect of crawl fixes?

Changes to robots.txt are picked up quickly, because Google generally caches the file for up to a day before fetching it again. Changes in how Googlebot spreads its requests take longer to show, typically several weeks, as it re-learns which URLs are worth visiting. Indexing improvements for pages that were previously stuck can take longer still. Compare Crawl Stats and log families four to six weeks after the change rather than checking daily, and remember that seasonal demand also moves crawl levels up and down.

See where your shop spends its crawl

The Ecommerce Crawl maps the URLs on your store, finds faceted navigation explosion and duplicate product URLs, flags out-of-stock products, surfaces orphans and measures how efficiently your crawl is spent.

Open Ecommerce Crawl Fixes →