Before your website can rank for anything — before any other SEO matters at all — Google has to be able to find your pages, read them, and add them to its index. If that foundation is broken, the best content in the world stays invisible. This is a plain-English, do-it-yourself guide to getting it right: no code, no jargon. It explains the three steps Google takes, how to make sure you’re not accidentally blocking it, and how to use Google’s own free tool to see exactly what it sees and fix what’s wrong.
It helps to know what actually happens behind the scenes. For a page to show up on Google, three things must happen in order. First, discovery: Google has to learn the page exists, usually by following a link to it or seeing it in your sitemap. Second, crawling: Google’s automated visitor, Googlebot, fetches and reads the page. Third, indexing: Google decides the page is worth storing and adds it to its index, the giant library it searches when someone types a query. A page only appears in results once it’s indexed — and each of these three steps can fail for reasons you can check and fix.
Every site has — or should have — a small plain-text file called robots.txt at the root of the domain (yoursite.com/robots.txt). It tells search engine crawlers which parts of the site they may and may not request. The single most common reason an entire website is invisible is a robots.txt file left blocking everything — often a leftover from when the site was being built, where a line like “Disallow: /” tells all crawlers to stay out.
To check yours, type your domain followed by /robots.txt into a browser. If you see Disallow: / with no path after the slash, that’s blocking your whole site and needs removing. A healthy basic robots.txt simply allows crawling and points to your sitemap. It’s also good practice to list your sitemap’s address inside robots.txt so crawlers find it immediately. Avoid blocking your CSS and JavaScript files, as Google needs them to understand the page properly. A quick Site Audit flags robots.txt problems and other crawl blockers in one go.
A sitemap (an XML sitemap) is a simple file listing all the pages you want Google to find — effectively a map of your site handed straight to the crawler. It doesn’t force Google to index anything; it’s a discovery and prioritisation signal that helps Google find your pages and notice when they’ve changed. The essential part of each entry is just the page’s address; an optional “last modified” date is useful when it’s accurate.
You usually don’t have to build this by hand. Most website platforms generate a sitemap automatically (WordPress does, via its built-in features or an SEO plugin), typically found at yoursite.com/sitemap.xml. One rule to follow: your sitemap and robots.txt must not contradict each other — never list a page in your sitemap that your robots.txt blocks, as that sends mixed signals.
Google Search Console is a free tool from Google, and it’s the single most useful thing you can set up — it shows you exactly how Google sees your site. Go to search.google.com/search-console, sign in with a Google account, and add your site as a “property.” You’ll prove you own it by following one of the on-screen verification methods (for example adding a small file or a DNS record — Search Console walks you through it). Once verified, you can see which pages are indexed, submit your sitemap, and find out why any page isn’t showing up.
Inside Search Console, go to the Sitemaps section, enter the address of your sitemap (such as sitemap.xml) and submit it. This tells Google where your map is so it can discover your pages faster. You only need to do this once; Google re-checks it over time. If your sitemap has errors, Search Console will tell you here.
This is where Search Console earns its keep. Two tools matter:
You’ll see statuses like these among the not-indexed pages. “Discovered – currently not indexed” means Google knows about the page but hasn’t crawled it yet, often a sign it’s waiting or sees the page as low priority. “Crawled – currently not indexed” means Google visited but decided not to store it, usually a quality or duplicate signal. “Blocked by robots.txt” points straight back to Step 1. And a noindex tag — a hidden instruction on a page telling Google not to index it — will keep a page out entirely; check for a stray one if an important page won’t index.
A quick free check any time, without Search Console, is to type site:yoursite.com into Google — it shows roughly what Google has indexed for your site. If almost nothing appears, you have a crawl or index problem worth chasing.
Imagine someone launches a small business site and, weeks later, can’t find it on Google at all. They type their domain plus /robots.txt and spot Disallow: / left over from the build — the whole site was blocked. They get that removed. They confirm their platform is generating a sitemap at /sitemap.xml. They set up Search Console, verify ownership, and submit the sitemap. A few days later the Page Indexing report shows pages moving from “Discovered” to indexed, and a search for their business name finally returns the site. The fix wasn’t content or keywords — it was unlocking the door and handing Google a map.
The big ones: leaving a “block everything” rule in robots.txt after launch (the number-one invisible-site cause). Forgetting to set up Search Console, so you’re flying blind. Having no sitemap, or one that contradicts robots.txt. Leaving a stray noindex tag on important pages. Blocking CSS and JavaScript so Google can’t render the page. And expecting instant results — a new site or page can take a few days to a couple of weeks to be crawled and indexed, so give it time before assuming something’s broken.
The most common reasons are a robots.txt file blocking crawlers, a noindex tag on your pages, the site being too new to be crawled yet, or Google not having found the pages. Set up Google Search Console and use URL Inspection to see the exact reason.
It’s a small text file at the root of your domain that tells search engine crawlers which pages they may and may not request. A stray “Disallow: /” in it can block your whole site.
A sitemap is a file listing the pages you want Google to find. It helps Google discover your pages, especially on newer or larger sites. Most website platforms generate one automatically.
Make sure robots.txt isn’t blocking you, create a sitemap, set up Google Search Console, submit your sitemap, and use URL Inspection to request indexing of important pages.
Anywhere from a few hours to a couple of weeks, depending on your site. New sites take longer. You can speed up important pages by requesting indexing in Search Console.
Type site:yoursite.com/your-page into Google for a quick check, or use the URL Inspection tool in Search Console for the accurate, official status.