AIWebPageSEO › Robots & Sitemap Fixes › Fix Robots.txt and Sitemap in Shopify

How to Fix Robots.txt and Sitemap in Shopify

Shopify generates robots.txt and sitemap.xml automatically and well — but since 2021 you can customise robots.txt via theme code. This guide covers Shopify-specific robots and sitemap management. Pair with robots/sitemap guide.

Step-by-step: How to fix Shopify robots.txt and sitemap

  1. Audit current robots.txt. Visit yourstore.com/robots.txt. Default Shopify robots.txt blocks: /admin, /cart, /checkout, /policies, /search, /apple-app-site-association, /account, and other operational paths. Most defaults are good.
  2. Customise via robots.txt.liquid. Online Store → Themes → Edit code → create templates/robots.txt.liquid. Copy default rules from Shopify docs, add custom rules. Format: '{{- 'now' | date: '%Y-%m-%d' -}}' for liquid; standard robots.txt syntax for rules.
  3. Configure AI bot policy. Add User-agent rules for AI agents. Allow GPTBot, ClaudeBot, PerplexityBot if you want AI visibility. Block training-only crawlers if you have copyright concerns.
  4. Block additional low-value paths. Common additions: '/search', URL parameters '/collections/*?filter_*', any vanity URLs not for indexing. Edit robots.txt.liquid carefully — typos break crawling entirely.
  5. Audit sitemap. Visit yourstore.com/sitemap.xml. Should include sub-sitemaps: sitemap_products_*.xml, sitemap_collections_*.xml, sitemap_pages_*.xml, sitemap_blogs_*.xml. Verify each has expected URL counts.
  6. Sitemap exclusion. Shopify auto-excludes draft products, draft collections, archived blog posts. Verify by checking expected URL count matches active content count. Excludes can't be customised without theme-level filtering.
  7. Submit to Search Console. Search Console → Sitemaps → Add 'sitemap.xml'. Repeat in Bing Webmaster Tools. Verify successful processing.
Tip. Document your Shopify configuration choices in a single internal doc (theme version, installed apps, custom code edits). When something breaks after a theme or app update, you have a baseline to compare against.

🤖 Audit Shopify robots/sitemap

Validate Shopify robots.txt rules and sitemap completeness.

Run Robots/Sitemap Audit →

Frequently Asked Questions

How do I edit Shopify robots.txt?

Online Store → Themes → Edit code. Create templates/robots.txt.liquid. Shopify reads this file and merges with defaults. Available since June 2021. Test by visiting yourstore.com/robots.txt after deploying.

Should I block AI bots in Shopify robots.txt?

Policy decision. Allow if you want AI search visibility (most retailers benefit from being cited in 'best X' AI queries). Block training-only bots (CCBot specifically) if you have copyright concerns about content training. Allow citation bots (GPTBot, ClaudeBot, PerplexityBot) regardless if you want AI search presence.

Why does Shopify's default robots.txt block /search?

Internal search results are low-value, duplicate, and create infinite crawl combinations. Default exclusion is correct. Don't unblock /search.

Can I exclude specific products or collections from Shopify sitemap?

Limited control. Setting product/collection to 'Draft' status excludes from sitemap. For published-but-excluded URLs, edit robots.txt.liquid to disallow specific paths. Or use a sitemap-control app.

Does Shopify sitemap regenerate automatically?

Yes — on every product/collection/page publish or update. The auto-generated sitemap at /sitemap.xml is always current. Submit it once to Search Console; subsequent updates are detected automatically.

Related Guides

Got a problem?