Robots.txt & Sitemap — Configuration Help

Get help configuring your robots.txt and XML sitemap. Share examples, discuss common mistakes, and help each other get crawling right.

Paste the relevant lines from your robots.txt and the address of your sitemap when you ask for help. Remember that robots.txt controls crawling, not indexing, so a blocked page can still be listed. Your sitemap should list only pages that load with a 200 status, are not set to noindex and point to themselves as canonical.