How to do an SEO site audit?
What a crawler actually does in a site audit
A site crawler is software that visits every page of your site in sequence, following links just as a search engine would, and records what it finds — each page's URL, title, description, heading structure, response code, load time, and links. The resulting report gives you an objective inventory of your entire site that would take days to compile manually. Most important: it shows you what a search engine actually sees, rather than what you think your site looks like in a browser. This distinction matters because a page can display perfectly visually but be fundamentally broken from a search engine's perspective.
Understanding HTTP status codes in an audit
One of the most practically useful outputs of a site crawl is the list of HTTP status codes returned by each URL. A 200 means the page loaded successfully. A 301 is a permanent redirect — useful when pages move, but chains of multiple redirects in sequence (302 → 301 → 200) waste what's called 'crawl budget' and dilute the authority passing through the redirect. A 404 means the page doesn't exist, which wastes any backlinks pointing to it. A 500 series code indicates a server error — more serious and potentially affecting how search engines view your site's reliability.
The XML sitemap check
Your XML sitemap is supposed to be a clean list of every URL you want search engines to index — but it frequently contains redirected URLs, noindexed pages, or pages returning 404 errors that weren't removed when pages were updated. A sitemap containing errors sends mixed signals to search engines: you're telling them a page is important by including it in the sitemap, while simultaneously telling them something is wrong with it. Cleaning the sitemap so it contains only clean, canonical, indexable URLs is a low-effort step that helps search engines allocate their crawl time to the pages that actually matter.
Key Takeaways
- A crawler shows what search engines actually see — not how the site looks in a browser.
- HTTP status codes reveal redirect chains, 404 errors, and server problems.
- Your XML sitemap should contain only clean, canonical, indexable URLs.