Fixing Shopify Crawl Issues

The URL families a Shopify store creates

Every Shopify store, whatever the theme, exposes a similar set of URL patterns. Knowing them turns a crawl report full of unfamiliar addresses into a short list of decisions.

The "blocked by default" column reflects Shopify's default file at the time of writing. Shopify updates its defaults from time to time, so always check your own live /robots.txt rather than relying on any list, including this one.

What the default robots.txt already does

Open https://your-store.example.com/robots.txt in a browser. The default file is long, with separate groups for all crawlers and for a few named bots, and it ends with a Sitemap: line. The rules that matter most for crawl waste block:

• Admin, cart, checkout, orders and account areas.
• Internal search results under /search.
• Collection URLs containing sort_by, so reordered listings are not crawled.
• Collection and blog URLs that combine tags with +.
• Collection URLs that combine two or more storefront filters.
• Policy pages under /policies/ and various internal endpoints.

What the defaults leave open is just as important. A single filter, a single tag, vendor and type listings, /collections/all, blog tag archives and paginated collections are all crawlable. On a small store that is usually fine. On a store with many collections, each with a dozen filter values and a long tag list, the single-value views alone can outnumber the products several times over. That is the gap robots.txt.liquid lets you close.

Editing robots.txt.liquid safely

Shopify lets you customise robots.txt by adding a robots.txt.liquid template to your theme: in the admin, go to Online Store → Themes, choose Edit code on the live theme, and add a new template of type robots.txt. The new template starts with Liquid that reproduces the default rules. Keep that structure and add to it. Shopify's developer documentation shows this pattern for adding a rule to the group for all crawlers:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /collections/vendors*' }}
    {{ 'Disallow: /collections/types*' }}
    {{ 'Disallow: /*/collections/vendors*' }}
    {{ 'Disallow: /*/collections/types*' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Because the loop outputs robots.default_groups, you keep every rule Shopify ships today and any it adds in future. Removing a default rule works the same way, with an unless test inside the rules loop that skips the rule you want to drop.

Rules worth considering

• Vendor and type lists, as in the example, if you do not use them as landing pages.
• Single filter values you have decided not to index, for example Disallow: /collections/*filter.v.price for price ranges, which are never useful as landing pages.
• Single tag views on older themes that filter by tag, if you have replaced the useful ones with proper collections.

Before blocking, remember that blocked pages cannot show Google a noindex tag. If a pattern is already indexed and you want it removed from results, use a noindex tag first, wait for the pages to drop out, then block. The general rules for choosing which filtered views to keep are in Fixing faceted URL explosion.

How the Shopify sitemap behaves

Shopify generates /sitemap.xml automatically as an index that links to child sitemaps for products, collections, pages and blog posts, split into several files on larger stores. You cannot edit it directly, but you can control what goes into it:

• Products not available on the Online Store sales channel are left out, as are unpublished pages and drafts.
• The seo.hidden metafield, set to 1 on a product, collection, page, article or blog, removes that resource from the sitemap and tells search engines not to index it. Shopify documents this for hiding pages such as thank-you pages or internal landing pages.
• Password-protected stores cannot be crawled at all, so a store in development or behind a password page will show nothing to search engines until the password is removed.
• Markets with subfolders or separate domains add further URLs; check which sitemap files are declared in your live robots.txt for each market rather than assuming.

Submit /sitemap.xml in Search Console once. Shopify keeps it current as products are added and removed.

Apps, themes and other hidden URL sources

Many surprise URLs on Shopify stores do not come from Shopify itself. Check these when the crawl finds patterns that are not in the table above:

• Theme product cards that link through collection paths. The fix is a small Liquid change, covered in the Shopify ecommerce SEO guide.
• Filter and search apps that replace Shopify's storefront filtering and use their own parameters, which the default rules do not know about.
• Review, loyalty and wishlist apps that add their own pages or proxy paths under /apps/ or /pages/.
• Currency and language switchers that append parameters to every internal link.
• Search result tracking parameters added to product links from on-site search, which are harmless because they carry a canonical, but inflate crawl reports.
• Removed apps that left snippets in the theme still outputting links to pages that no longer exist.

The fix, step by step

What you cannot change on Shopify

Some things are fixed by the platform, and it saves time to know them before planning work:

• You cannot change the /products/ and /collections/ prefixes or remove them from URLs.
• You cannot stop the collection-path product URLs from existing; you can only stop linking to them.
• You cannot read raw server logs, so Crawl Stats and crawl tools are your evidence.
• You cannot set HTTP status codes for individual pages; removed products return Shopify's standard 404 page, and redirects are managed in the admin's URL redirects section rather than on the server.
• You cannot edit the XML sitemap by hand, only influence what goes into it.

Server capacity is rarely the limit on Shopify, because hosting is managed and generally fast. Crawl problems on Shopify stores are almost always about how many URLs exist, not how quickly they load. For broader background, the Shopify robots and sitemap fixes cover syntax errors and sitemap warnings.