Fixing WooCommerce Crawl Issues
WooCommerce itself creates a fairly tidy set of URLs, but a working shop adds a lot on top: add-to-cart links on every product card, filter and price parameters from the sidebar, sort orders, product search, and whatever URLs your wishlist, compare and filter plugins bring.
Crawlers follow many of them, and on a shop with a few hundred products they can end up requesting tens of thousands of addresses. This guide lists the WooCommerce URL patterns that cause crawl waste, shows how to find which ones are actually being crawled, and gives the template, plugin and robots.txt changes that bring the crawl back to your products and categories.
The URL patterns a WooCommerce shop exposes
Start by knowing what to look for. These are the patterns that most often turn up in the Ecommerce Crawl results and in server logs for WooCommerce shops. The parameter names below are WooCommerce defaults; plugins may use their own.
| Pattern | Where it comes from | Should crawlers fetch it? |
|---|---|---|
?add-to-cart=123 | "Add to basket" buttons on shop and category pages for simple products | No; it is an action, not a page |
?filter_colour=red&query_type_colour=or | Attribute filters in the classic layered navigation widget and similar blocks | Mostly no; a few combinations may deserve landing pages |
?min_price=20&max_price=50 | Price filter | No |
?rating_filter=4 | Rating filter | No |
?orderby=price-desc | Sort dropdown on shop and category pages | No |
?s=boots&post_type=product | Product search | No |
/product-category/boots/page/4/ | Pagination | Yes, so products deeper in the list are found |
/cart/, /checkout/, /my-account/ | WooCommerce system pages | No need; they have no search value |
?add_to_wishlist=, compare and quick-view parameters | Wishlist, compare and quick-view plugins | No |
?wc-ajax= | WooCommerce AJAX endpoints used by scripts | No, but do not block it for users |
The trouble is multiplication. One category with five colour options, four sizes, a price slider and four sort orders can produce hundreds of combinations, and each filter link on each of those pages leads to more. Add-to-cart links are a separate problem: every product card on every one of those filtered pages carries one.
Structural choices such as permalinks, tags and attribute archives are covered in the WooCommerce SEO guide. This page stays with crawling.
Finding out what is actually crawled
A pattern existing on your site does not mean it is being crawled heavily. Before writing rules, check what search engines really request.
Search Console Crawl Stats
Under Settings, Crawl stats shows total requests over time, a breakdown by response code, file type and purpose, and example URLs. A sudden rise in requests with no new products, or example URLs full of parameters, points to a trap.
Server access logs
Logs show every request. On Plesk, each domain's logs are under Logs in the domain panel; on cPanel
under Raw Access; managed WordPress hosts usually offer a download. Filter for requests whose user agent
contains Googlebot, then group by the part of the URL before the first value, for example
counting how many requests contain add-to-cart=, filter_, orderby=
or min_price=. A spreadsheet or a short command line script is enough.
grep "Googlebot" access_log | grep -o "[?&][a-z_-]*=" | sort | uniq -c | sort -rn | head -20That one-liner counts which query parameter names Googlebot is requesting most. Anyone can put "Googlebot"
in a user agent, so for anything important confirm the requests come from Google using a reverse DNS
lookup, which should resolve to a googlebot.com or google.com host.
Add-to-cart links in detail
WooCommerce renders the "Add to basket" button for simple products on archive pages as a real link
with an add-to-cart parameter, so the button still works without JavaScript. WooCommerce
adds rel="nofollow" to those links, but nofollow is treated as a hint, and crawlers can
still discover and request the URLs. Each request can also run cart logic on your server, and if page
caching skips URLs with that parameter, every one hits PHP.
Your options, from least to most effort:
- Disallow the parameter in robots.txt with
Disallow: /*add-to-cart=. This matches it whether it appears after?or&. - Check that your theme is not linking to add-to-cart URLs from places that do not need them, such as related-product carousels on every page.
- Some themes and plugins can render the archive button as a
<button>that submits via AJAX rather than a link, which removes the URL from the HTML entirely.
/wp-admin/admin-ajax.php or ?wc-ajax=
for users with server rules. They power the basket and checkout. robots.txt only affects well-behaved
crawlers, not shoppers, so disallowing crawler access there is safe; firewall rules are not.Filters: choose before you block
Some filtered views are worth having in search. "Women's red walking boots" might be a phrase with
real demand that no core category covers. If so, the better answer is usually a proper sub-category with
its own clean URL and introduction, rather than an indexable ?filter_colour=red page. The
general method for choosing is covered in Fixing
faceted URL explosion.
For WooCommerce specifically, check what your filter output looks like:
- Canonical. View source on a filtered category URL. Most SEO plugins set the canonical to the unfiltered category. If yours points at the filtered URL itself, every combination claims to be the original.
- Links. Many filter plugins can apply filters by JavaScript without generating crawlable
links, or add
nofollow. Look in the plugin's settings for options that change how filter links are output. - Plugin parameters. Third-party filter plugins often use their own parameter names
rather than
filter_. Your robots.txt rules must match what your site actually uses, so take the names from real URLs.
The order matters. If filtered pages are already indexed, let them be crawled until canonical or noindex has taken effect and they have dropped out, then add robots.txt rules. A blocked page cannot have its noindex read.
Cart, checkout, account and search
WooCommerce outputs a noindex robots meta tag on the cart, checkout and my account pages, so they should not be indexed even if crawled. You can additionally disallow them in robots.txt to save requests, since there is nothing on them to consolidate. If your theme or SEO plugin has changed these pages, view the source of each and confirm a noindex tag is present.
Product search results are a common source of endless URLs, because every search term creates a new
address and some spam bots deliberately create searches that get linked. Most SEO plugins noindex search
results; Google advises not letting internal search pages be indexed. A Disallow: /*?s=
rule stops crawling of them, but note it also covers blog search on the same WordPress site, which is
usually what you want anyway.
Yoast SEO, Rank Math and similar plugins include options for removing some WordPress extras such as certain feed links, and some can strip unrecognised URL parameters. The exact options vary by plugin and version, so check the crawl or advanced settings in yours and test after changing anything.
A starting robots.txt for WooCommerce
WordPress serves a virtual robots.txt unless a physical robots.txt file exists in the
site root, in which case the file wins. Yoast SEO and Rank Math both include a robots.txt editor. Whichever
you use, adapt this starting point to the parameter names your shop really uses:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=
Disallow: /*orderby=
Disallow: /*min_price=
Disallow: /*max_price=
Disallow: /*rating_filter=
Disallow: /*?s=
Disallow: /*add_to_wishlist=
Sitemap: https://www.example.co.uk/sitemap_index.xmlNotes on this example:
- It does not block
filter_parameters, because you may want some filtered views crawled while canonicals settle. AddDisallow: /*filter_once you have decided none are needed. - Paths like
/cart/assume default page slugs. If your basket page is/basket/, use that. - Never disallow
/wp-content/or/wp-includes/. Search engines need the CSS, JavaScript and images there to render your pages. - The sitemap line should match whatever your SEO plugin generates.
Test each rule before publishing. Search Console's robots.txt report shows the version Google fetched and any parse problems, and URL Inspection shows whether a specific URL is blocked. See the WordPress robots and sitemap guide for the WordPress side in more depth.
The fix, step by step
Step 1
Find which URL patterns Googlebot requests
Run the Ecommerce Crawl to list the URL families your shop exposes, then download a month
of server access logs and count Googlebot requests by query parameter name. Check Search Console Crawl
stats for example URLs and response codes. You want a ranked list of patterns such as add-to-cart=,
filter_, orderby= and plugin parameters, showing where crawling is actually
going, so the rules you write later target real waste rather than guesses.
Step 2
Stop add-to-cart and action links being followed
Add Disallow: /*add-to-cart= and equivalent rules for wishlist, compare
and quick-view parameters to robots.txt. Check whether your theme offers an AJAX add-to-cart button on
archives that avoids a link altogether, and remove add-to-cart buttons from carousels where they add
nothing. Never block admin-ajax.php or wc-ajax for real visitors, because the
basket, mini-cart and checkout all depend on them to update without reloading the page.
Step 3
Decide which filter parameters may be indexed
List the filter parameters your shop uses, taken from real URLs rather than assumptions, since plugins often rename them. For any combination with genuine search demand, create a proper sub-category instead. For the rest, confirm filtered pages carry a canonical to the unfiltered category, and use your filter plugin's settings to stop outputting crawlable links for filters where it offers that option, which keeps the combinations from being discovered at all.
Step 4
Neutralise sort, view and search URLs
Check that sorted and paginated-per-page views canonicalise to the default category URL,
and that product search results carry noindex from your SEO plugin. Once those are in place, add robots.txt
rules for orderby=, min_price=, max_price=, rating_filter=
and ?s=. Leave normal pagination crawlable so products listed further down each category
can still be found, and do not add noindex to those paginated pages either.
Step 5
Check the cart, checkout and account pages
View the source of your cart, checkout and my account pages and confirm each carries a noindex robots meta tag, which WooCommerce adds by default unless a theme or plugin has removed it. Confirm the real slugs of those pages under WooCommerce settings, then disallow them in robots.txt to save crawl requests. Make sure none of them appear in your XML sitemap or in any menu that crawlers follow.
Step 6
Write and test the robots.txt rules
Put
the rules in your SEO plugin's robots.txt editor or a physical file in the site root, remembering a physical
file overrides the WordPress virtual one. Keep /wp-content/ and /wp-includes/
open. After publishing, open the robots.txt report in Search Console to confirm Google fetched the new
version, and use URL Inspection on a product, a category and a blocked parameter URL to check each is
treated as intended.
Step 7
Recheck after plugin and theme updates
New filter, wishlist, currency and translation plugins often add parameters of their own. After installing or updating any of them, view a category page's links and rerun the Ecommerce Crawl to spot new URL families. Review your logs again a month after the changes; Googlebot requests to blocked patterns should fall, with more of its crawling going to product and category URLs that you actually want found.
1. Why is Google crawling my add-to-cart URLs?
WooCommerce renders the add to basket button on shop and category pages as a real link containing an add-to-cart parameter, so it works without JavaScript. Although WooCommerce marks those links nofollow, crawlers treat nofollow as a hint and may still request them, and every filtered or sorted category page multiplies the number of links. Disallowing the parameter in robots.txt stops compliant crawlers fetching them, and an AJAX-only button avoids the link entirely.
2. Where do I edit robots.txt on a WooCommerce site?
WordPress generates a virtual robots.txt automatically. SEO plugins such as Yoast SEO and Rank Math include an editor for it in their tools or general settings. Alternatively, you can upload a physical robots.txt file to the site root with your host's file manager, and that file then replaces the virtual one completely. Only use one method, so you are never unsure which version is live, and check the result in Search Console's robots.txt report.
3. Should I block filter_ parameters in robots.txt?
Eventually, for most shops, but not first. If filtered pages are already indexed, they need to be crawled so search engines can read their canonical or noindex and drop them. Block too early and they can linger in results. Decide first whether any filtered view deserves to be a proper sub-category, make sure the rest canonicalise to the main category, and add a robots.txt rule once they have dropped out of the index.
4. Does blocking cart and checkout in robots.txt affect customers?
No. robots.txt is read only by crawlers that choose to follow it, such as Googlebot and Bingbot. Browsers ignore it completely, so customers can still use the basket, checkout and account pages as normal. The one thing to avoid is using server or firewall rules to block those paths or the AJAX endpoints, because those would affect real visitors. WooCommerce also marks these pages noindex, so they should not appear in search either way.
5. How can I tell if crawl budget is actually a problem on my WooCommerce shop?
Compare what is crawled with what exists. Count your products, categories and content pages, then look at how many distinct URLs Googlebot requests in a month from your server logs, or at the examples in Search Console Crawl stats. If crawled URLs are many times your real page count and are mostly parameters, crawling is being wasted. If new products take a long time to be crawled and indexed, that points the same way.
6. Will these changes speed up my WooCommerce site?
They can reduce server load. Parameter URLs such as add-to-cart and filters are often excluded from page caching, so each crawler request runs PHP and database queries. Cutting thousands of those requests a day frees resources for real shoppers. They will not change how fast a single page loads for a visitor, which depends on hosting, theme, images and scripts. Treat crawl control and page speed as related but separate jobs.
7. My filter plugin uses different parameter names. What should I block?
Block the names your site actually produces. Open a category page, apply a few filters and a sort order, and copy the resulting URLs; the parameter names are the parts before each equals sign. Plugins may use a single parameter for all filters or a separate one per attribute. Check your server logs as well, since crawlers sometimes find older parameter formats from previous plugins, which also need handling or redirecting.
The Ecommerce Crawl maps the URLs on your store, spots faceted navigation getting out of hand, finds duplicate product URLs and measures how efficiently your crawl is spent.
Open Ecommerce Crawl Fixes →