Site Crawler Fixes: Clearing the Path for Search Engine Bots
A search engine learns about your site by starting at a known URL, fetching it, reading the links, and repeating.
The Site Crawler follows the same process on up to 10,000 URLs and records what a bot would run into along the way: links that lead nowhere, redirects that take several hops, pages nothing links to, near-identical URLs, sections blocked by robots rules, sitemaps that disagree with the real site, templates that respond slowly and pages with very little on them. This page explains how those findings connect and which guide fixes each one.
How a crawl sees your site
It helps to picture a crawl as a journey rather than a checklist. The crawler arrives at your homepage, reads every link, adds the new ones to a queue, and works through the queue. Each finding in the report is a point where that journey went wrong or was wasted.
- Dead ends: a link leads to a 404 or other error. The crawler spent a request and learned nothing useful.
- Detours: a link leads to a redirect, which leads to another redirect before reaching a real page. Each hop is an extra request and extra delay.
- Locked doors: robots.txt or a meta robots tag says a section is off limits, sometimes by accident.
- Islands: pages that exist, and may be in the sitemap, but that no link on the site reaches. A link-following crawler never arrives there.
- Hall of mirrors: the same content on several URLs, so the crawler fetches it repeatedly and search engines must decide which copy counts.
- Slow queues: templates that take a long time to respond, which can lead search engines to fetch fewer pages.
- Empty rooms: pages with little or no useful content, which use crawl effort and dilute the overall quality of what is indexed.
For a site of a few hundred pages, search engines can usually crawl everything regardless. The effects become more serious on large sites, and on sites where the waste is severe, such as endless filter combinations. But even on a small site, dead ends and islands hurt visitors as much as bots, so they are always worth fixing.
The fix guides
Each guide takes one type of finding, explains the causes, and gives a fix with platform notes where they differ.
Links that go wrong
Fixing broken internal and external links
covers deciding between redirecting, restoring or removing each 404, cleaning up after a migration, and
redirect patterns for Plesk, cPanel and .htaccess. Fixing
redirect chains and loops covers collapsing multi-hop redirects into one, spotting loops, and what
to do when a CDN and the origin server both try to redirect the same request.
Pages the crawl cannot reach properly
Fixing orphan pages and crawl depth
covers finding pages with no internal links, choosing whether to link them in or retire them, and keeping
important content close to the homepage. Fixing
accidental robots blocks covers Disallow rules and noindex tags left over
from staging, and how to confirm the fix in Search Console.
URLs that confuse the index
Fixing duplicate content and canonicalisation covers query-string variants, pagination and printer versions, and when to use a canonical tag rather than a redirect. Fixing sitemap and crawl mismatches covers sitemap entries that return errors or redirects, real pages missing from the sitemap, and generating the file automatically.
Pages that waste crawl effort
Fixing slow templates that throttle the crawl covers using the crawler's timing data to find slow page types and fixing the query or rendering step responsible. Fixing thin content and low-value pages covers near-empty tag and category archives and when to improve, merge, noindex or delete them.
Which findings to fix first
Crawl findings interact, so the order matters more here than in most reports. Fixing one can create or remove another.
- Robots blocks on pages you want found. Nothing else matters for a blocked section. This is also the quickest fix, often a single line.
- Broken internal links. They are fully under your control, visible to every visitor, and each one fixed removes a dead end.
- Redirect chains and loops. Loops make pages unreachable. Chains are usually created by several rounds of changes, so fix them after broken links, which sometimes get redirected as a quick patch.
- Duplicate URLs. Decide the one true URL for each piece of content before you start adding internal links, or you may link to the wrong version.
- Orphan pages and depth. Once the right URLs are settled, link them into the structure.
- Sitemap mismatches. The sitemap should reflect the finished structure, so update it after the steps above.
- Thin pages and slow templates. These take more thought and often need content or development time, so plan them as a separate project.
Platform notes
The same finding often has a platform-specific cause. A few patterns come up again and again.
WordPress
The "Discourage search engines from indexing this site" option under Settings, Reading is a frequent source of sitewide blocks after launch. SEO plugins such as Yoast and Rank Math generate the sitemap and canonical tags, so check their settings before editing templates by hand. Tag and author archives with one or two posts are a common source of thin pages, and changing permalink settings is a common source of broken links.
Shopify
Products can be reached through collection paths such as /collections/shoes/products/trainer
as well as the main /products/trainer URL. Shopify sets the canonical to the main product
URL, but themes that link to the collection path create extra URLs for crawlers to fetch. Filtered collection
pages can multiply quickly. Shopify generates the sitemap for you and allows robots.txt changes through
a theme template, so some fixes are done differently from a self-hosted site.
Static site generators and frameworks
With Hugo, Jekyll, Eleventy, Astro or Next.js, the sitemap and canonical tags are usually produced at build time, so fixes go in the build configuration. Renaming a content file silently changes its URL, which is the typical cause of broken links after a rebuild. Adding a link check to the build or deployment pipeline catches these before they go live.
Reading crawl data alongside other sources
The crawler shows what a bot would find by following links from your homepage. That is one view, and it is most useful when compared with two others.
| Source | What it tells you | What the comparison reveals |
|---|---|---|
| Site Crawler | Every URL reachable through links, with status codes, redirects, depth and timing. | The structure as bots and visitors actually experience it. |
| XML sitemap | The URLs you have told search engines are important. | URLs in the sitemap but not found by the crawl are likely orphans; URLs found by the crawl but not in the sitemap may be missing or may be duplicates. |
| Search Console | What Google has actually crawled and indexed, and why some pages are excluded. | Whether Google's picture matches yours, and which crawl problems it has already met. |
For example, if the crawler finds 400 URLs, the sitemap lists 350 and Search Console reports 900 known pages, the extra URLs Google knows about are often parameter variants, old URLs that still receive links from elsewhere, or pages from a previous version of the site. That gap is where to look for duplicates and redirect work. The example crawl report shows how a clean crawl looks for comparison.
Order of work
- Run the Site Crawler from your homepage and save the results as a baseline.
- Check robots.txt and meta robots on any section that returned no pages or fewer pages than expected.
- Correct broken internal links at their source, in menus, templates and content.
- Replace chained redirects with single hops, and update internal links so they point at final URLs rather than relying on redirects.
- Choose canonical URLs for duplicate content and apply canonical tags or redirects.
- Link orphan pages into relevant sections, or retire them.
- Regenerate the sitemap so it contains only live, indexable, canonical URLs, and resubmit it in Search Console.
- Plan work on slow templates and thin pages, then crawl again and compare with the baseline.
If you are new to crawling, the beginner tutorial walks through a first crawl step by step, and the Site Crawler Guide is the full reference for every check. For on-page problems such as titles and headings, which the crawler is not designed to judge, use the Site Audit fixes.
1. What is crawl budget and should a small site worry about it?
Crawl budget is a loose term for how many URLs a search engine is able and willing to fetch from your site in a given period. Google has said that most sites with a modest number of pages do not need to worry about it, because their pages are crawled efficiently anyway. It becomes a real concern on very large sites, or where a site generates huge numbers of URLs through filters, calendars or parameters that waste the crawler's time.
2. Why does the crawler find pages that are not in my sitemap?
Usually because links on the site point to URLs you did not intend to be separate pages, such as filtered or sorted listings, tracking parameters, pagination or old URLs that are still linked from content. Sometimes it is simply because the sitemap is out of date and is missing real pages. Check each group: add genuine pages to the sitemap, and for variants either stop linking to them or use canonical tags pointing at the main version.
3. Why are some pages in my sitemap not found by the crawler?
The crawler follows links from your homepage, so a page it cannot reach through links is effectively an orphan, even if it is listed in the sitemap. Search engines can still discover it through the sitemap, but it receives no internal links to show its importance. Decide whether each page is still wanted. If it is, link to it from relevant pages. If it is not, remove it from the sitemap and redirect or retire the URL.
4. Should I fix broken links with redirects or by editing the links?
Edit the links wherever you control them. A redirect is a useful safety net for visitors arriving from bookmarks or other sites, but relying on redirects for your own internal links adds an extra request for every click and can build into chains over time. The clean approach is to redirect the old URL to its best replacement and also update every internal link to point straight at the new URL.
5. How long does it take Google to notice crawl fixes?
It varies with how often Google visits each page. Popular pages such as the homepage may be recrawled within days, while deeper pages can take weeks. You can speed things up for a few important URLs with the URL Inspection tool in Search Console, and by resubmitting an updated sitemap. Watch the Pages and Crawl stats reports in Search Console over the following weeks to see the number of errors and redirects fall.
6. Is it bad to have lots of redirects on my site?
Having redirects is normal, especially after a redesign or migration, and a single 301 from an old URL to its replacement is the right thing to do. The problems are chains, where one redirect leads to another, and loops, where redirects point back at each other. Internal links that go through redirects are also avoidable waste. Keep each old URL redirecting in one hop to its final destination, and link directly to final URLs within your own site.
7. Can I stop search engines crawling thin pages with robots.txt?
You can, but it is often not the best tool. A Disallow rule stops crawling, not indexing, so a blocked URL can still appear in results if other pages link to it, and search engines cannot see any noindex tag on it. For thin pages you want out of the index, a noindex tag that crawlers can read is usually better. For pages with no value at all, deleting them or merging them into a stronger page is cleaner still.
The Site Crawler follows your links on up to 10,000 URLs and reports broken links, redirect chains, orphans, duplicate URLs, robots blocks, sitemap mismatches, slow templates and thin content.
Run the Site Crawler →