A site crawl visits every page on your website — just like Google does — and records every SEO issue it finds. Missing title tags, broken internal links, thin content, duplicate pages — a crawl finds them all in one pass before Google penalises you for them.
🕷 Crawl Your Site All Performance Tools →When Google visits your site, it starts at your homepage and follows every internal link it finds — visiting page after page, recording what it finds on each one. A site crawl replicates this process and shows you what Google sees: which pages are accessible, which have errors, and which have SEO issues that could be costing you rankings.
| Issue | Priority | Why |
|---|---|---|
| Pages returning 404 or 500 | 🔴 Critical | Wastes crawl budget, harms user experience |
| Missing title tags | 🔴 Critical | Google writes its own — usually worse than yours |
| Missing H1 tag | 🟠 High | Weakens topical relevance signal |
| Duplicate title tags | 🟠 High | Confuses Google about which page to rank |
| Thin content under 300 words | 🟡 Medium | May be skipped in indexing |
| Missing meta descriptions | 🟡 Medium | Google writes its own snippet |
| Broken internal links | 🟠 High | Wastes link equity, frustrates users |
The premise — "a crawl replicates what Google does" — is right in outline and wrong in four specific ways that determine whether your report is trustworthy.
Google indexes with a mobile user-agent. Whatever is on your desktop layout is not what is being assessed. If a section of content is hidden, truncated or simply not rendered on mobile, Google may never see it — and a desktop crawl will report it as present and fine. Any crawl you rely on should be run as mobile, or you are auditing a version of your site that Google does not index.
Googlebot executes JavaScript, but not immediately — rendering is queued and can happen well after the initial fetch. A crawler that does not render sees the raw HTML: on a client-rendered site, that is an empty shell. It will report no missing titles, no thin content and no broken links, because it found no content at all to have problems with. A suspiciously clean report on a JavaScript-heavy site is not good news; it is a crawler seeing nothing.
Googlebot adjusts its crawl rate according to how your server responds. Slow responses and 5xx errors cause it to back off. So an overloaded server does not just serve errors: it reduces the rate at which Google is willing to visit at all, which slows discovery of everything you publish afterwards. This is a real feedback loop and it is why server health is an SEO issue rather than an infrastructure one.
Googlebot knows URLs from old sitemaps, expired links, past crawls and other people's pages. It will revisit a URL you deleted and unlinked years ago. Your crawler starts from the homepage with no history. Google's picture of your site is larger than yours, and the gap is where the surprises live.
If a crawl runs away — consuming its page budget on URLs you have never heard of, never reaching the pages you cared about — you have found a crawl trap. The important part is that Googlebot is in the same trap, spending its crawl allowance on the same worthless URLs, which is why your new pages take a fortnight to be indexed.
The usual sources, in order of how often they occur:
?sort=price&view=grid&per_page=48 — every permutation is a URL, and the content is identical./a/b/a/b/a/b/…, which resolves happily and forever.noindex prevents a crawl — both live inside the page, so the crawler has to fetch the page to read them. They control indexing, not crawling. Only robots.txt, and not generating the crawlable link in the first place, stop the fetch. A site with noindex on a hundred thousand filter URLs still has all hundred thousand crawled.A crawler follows <a href> links from a starting point. That single rule defines a large set of blind spots, and they are systematic rather than occasional.
<div onclick>, a <button> that navigates, an <a href="javascript:void(0)">. These are invisible. If your category menu is built this way, your entire product tree is unreachable.Every one of these is a Google blind spot too. A crawl that cannot reach a page is telling you something important: neither can the search engine.
Your crawl tells you what a bot could fetch. Search Console tells you what Google decided. Your server logs tell you what Googlebot actually did — and it is the only one of the three that is a record of events rather than an inference.
What the logs reveal that nothing else can:
Search Console's Crawl Stats report is a usable substitute if raw logs are out of reach — it shows response codes, response times and what Googlebot spent its time on. It is less precise and vastly better than guessing.
robots.txt, and know when to ignore it deliberately. The default should be to obey — you want to see what Google sees. But a one-off crawl that ignores robots.txt is the fastest way to discover that you have accidentally blocked a section you needed indexed, because it will appear in the crawl and be missing from the index.Retry-After header. It is the correct, temporary, understood signal. Blocking Googlebot in robots.txt to "give the server a rest" causes pages to drop out of the index, and getting them back is far more expensive than the outage you were avoiding.A site crawl checks every page it can reach from your homepage by following internal links. For each page it records: the HTTP status code (200, 404, 500 etc), the title tag and meta description, heading structure (H1, H2), page speed, internal and external links, canonical tags, robots meta tags, and structured data. The result is a complete technical SEO audit of your entire site.
The aiwebpageseo Site Crawler takes approximately 30-60 seconds per 10 pages crawled. A 50-page crawl typically completes in 3-5 minutes. The crawler respects your server resources and adds a small delay between requests to avoid impacting site performance.
Monthly for small sites. Weekly for active sites that publish regularly or have frequent content updates. Always run a full crawl immediately after any site migration, CMS update, redesign or significant structural change. Any of these events can introduce new technical issues.
You have a crawl trap — and Googlebot is in it too, which is why new pages take weeks to be indexed. The usual causes are faceted navigation generating every filter combination as a distinct URL, a calendar widget whose "next month" link runs to the year 2400, session IDs or tracking parameters that mint a fresh URL on every visit, and sort or view parameters producing identical content at endless addresses. Note that neither a canonical nor a noindex prevents the crawl, since both must be fetched to be read — only robots.txt and not generating the link at all will stop it.
Not necessarily. On a JavaScript-heavy site, a crawler that does not render JavaScript sees an empty shell — so it reports no missing titles, no thin content and no broken links because it found no content at all to have problems with. Check whether your crawl renders. A suspiciously clean report on a client-rendered site usually means the crawler saw nothing.
Mobile. Google indexes with a smartphone user-agent, so the desktop layout is not the version being assessed. Content that is hidden, truncated or not rendered on mobile may never be seen by Google, while a desktop crawl reports it as present and fine — meaning you would be auditing a version of your site that Google does not index.
Because it follows links from a starting point, so it reaches only what is linked. Pages behind a login, pages reachable only by submitting a form, navigation built from divs or buttons rather than real anchors, infinite scroll with no paginated URLs, and orphan pages linked from nowhere are all invisible to it. Each of these is a Google blind spot as well — if a crawler cannot reach a page, neither can the search engine.
Run the Site Crawler and get actionable results in minutes. Pay as you go — no subscription needed.
Crawl Your Site →aiwebpageseo.com is a data-driven SEO and AEO (Answer Engine Optimisation) platform providing a free suite of technical website tools. Rather than relying on AI-theorised assumptions, the platform analyses live URL performance, delivering objective diagnostics, page speed metrics, CLS debugging, and site crawl data alongside actionable technical tutorials.