How to do an SEO site audit?
What a crawler actually does in an SEO site audit
In an SEO site audit, a site crawler is software that visits every page of your site in sequence, following links just as a search engine would, and records what it finds — each page's URL, title, description, heading structure, response code, load time, and links.
The resulting report gives you an objective inventory of your entire site that would take days to compile manually. Most important: it shows you what a search engine actually sees, rather than what you think your site looks like in a browser. This distinction matters because a page can display perfectly visually but be fundamentally broken from a search engine's perspective.
Understanding HTTP status codes in an audit
One of the most practically useful outputs of a site crawl is the list of HTTP status codes returned by each URL. A 200 means the page loaded successfully. A 301 is a permanent redirect — useful when pages move, but chains of multiple redirects in sequence (302 → 301 → 200) waste what's called 'crawl budget' and dilute the authority passing through the redirect. A 404 means the page doesn't exist, which wastes any backlinks pointing to it. A 500 series code indicates a server error — more serious and potentially affecting how search engines view your site's reliability.
The XML sitemap check
Your XML sitemap is supposed to be a clean list of every URL you want search engines to index — but it frequently contains redirected URLs, noindexed pages, or pages returning 404 errors that weren't removed when pages were updated. A sitemap containing errors sends mixed signals to search engines: you're telling them a page is important by including it in the sitemap, while simultaneously telling them something is wrong with it. Cleaning the sitemap so it contains only clean, canonical, indexable URLs is a low-effort step that helps search engines allocate their crawl time to the pages that actually matter.
Key Takeaways
- A crawler shows what search engines actually see — not how the site looks in a browser.
- HTTP status codes reveal redirect chains, 404 errors, and server problems.
- Your XML sitemap should contain only clean, canonical, indexable URLs.
Related Questions
Frequently asked questions
What does a site crawler do during an SEO site audit?
A site crawler is software that visits every page of a website in sequence, following links in the same way a search engine would. It records each page's URL, title, description, heading structure, response code, load time and links. The resulting report gives an objective inventory of the whole site that would take days to compile by hand. Most importantly, it shows what a search engine actually sees rather than how the site looks in a browser. That matters because a page can display perfectly to a visitor while being fundamentally broken from a search engine's point of view.
What do HTTP status codes mean in a site audit?
Status codes show how each URL responds. A 200 means the page loaded successfully. A 301 is a permanent redirect, which is useful when pages move, but chains of several redirects in sequence waste crawl budget and dilute the authority passing through them. A 404 means the page does not exist, so any backlinks pointing to it are wasted. A code in the 500 series indicates a server error, which is more serious and can affect how search engines view the reliability of your site. Reviewing these codes across every URL is one of the most practically useful outputs of a crawl.
Why should an XML sitemap only contain clean URLs?
An XML sitemap is meant to be a clean list of every URL you want search engines to index, yet it often contains redirected URLs, noindexed pages or pages returning 404 errors that were never removed. A sitemap with errors sends mixed signals: including a page says it is important, while its status says something is wrong with it. Cleaning the sitemap so it lists only canonical, indexable URLs that load properly is a low-effort task. It helps search engines spend their crawl time on the pages that actually matter rather than on broken or redirected addresses.