/ โ† Robots and Sitemap Fixes / Create an XML sitemap
In this guide
  1. What an XML sitemap is for
  2. A valid XML sitemap example
  3. How to find the XML sitemap on a site
  4. Ways to create an XML sitemap
  5. How to check an XML sitemap before submitting
  6. A worked example
  7. The fix, step by step
  8. Edge cases worth knowing
  9. Keeping the sitemap accurate
  10. Questions and answers

What an XML sitemap is for

An XML sitemap is a file that lists the URLs on your site you want search engines to know about, optionally with the date each one last changed. It helps crawlers discover pages, especially new pages, pages with few internal links, and large sites where some sections are deep in the structure. It does not force anything into the index: Google still decides what to crawl and index, and a page in your sitemap can still be left out if it is thin, duplicated or blocked.

The format is defined by the sitemaps.org protocol, which Google, Bing and other engines all support. The limits that matter:

RuleDetail
Size per fileUp to 50,000 URLs and 50 MB uncompressed; split into several files and use a sitemap index beyond that
EncodingUTF-8, with special characters escaped, for example & for an ampersand
URLsFull absolute addresses, on the same host and protocol as the site
lastmodUsed by Google when it is consistently accurate; set it when the page content really changes
changefreq and priorityIgnored by Google; harmless but not worth maintaining

A sitemap should contain only canonical URLs that return 200 and that you want indexed. Redirects, 404s, noindexed pages and parameter duplicates do not belong in it, and a sitemap full of them makes the whole file less trustworthy as a signal.

A valid XML sitemap example

This is a complete, valid sitemap for a small site. You can copy it as a starting point and change the addresses:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/menu/cakes</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/contact</loc>
  </url>
</urlset>

Each page sits in its own <url> element with a <loc>. The lastmod value uses the W3C date format, either a date or a full date and time with a time zone such as 2026-10-01T09:30:00+01:00.

A sitemap index for larger sites

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-pages.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
  </sitemap>
</sitemapindex>

Submit the index and search engines read every child file listed in it. Splitting by type, such as pages, posts and products, also makes the Search Console report easier to read. See sitemap size limits if you are close to the caps.

How to find the XML sitemap on a site

There is no single required address, so check these in order:

  1. robots.txt. Open https://example.com/robots.txt and look for a Sitemap: line. This is the most reliable clue, because it is where sites declare their sitemap.
  2. Common addresses. Try /sitemap.xml, /sitemap_index.xml (Yoast and Rank Math), /wp-sitemap.xml (WordPress core) and /sitemap-index.xml.
  3. Search Console. For your own site, the Sitemaps report lists every sitemap submitted for the property and when Google last read it.
  4. Your platform. Shopify publishes /sitemap.xml automatically, and most CMS SEO plugins show the address on their settings screen.

From a terminal you can check the status and the type of file the server returns:

curl -sI https://example.com/sitemap.xml | grep -iE '^(HTTP|content-type)'
curl -s https://example.com/robots.txt | grep -i '^sitemap'

A healthy result is a 200 with a content type such as application/xml or text/xml. If you get an HTML page, the address is probably being caught by a 404 page or a redirect.

Ways to create an XML sitemap

MethodBest forWatch out for
CMS or plugin (WordPress, Shopify, Wix, Squarespace)Most small business sites; updates itselfNoindexed or thin archive pages being included by default
Framework code (Next.js, Nuxt, Astro)Custom builds; generated from your real routesForgetting dynamic pages such as products
Online generatorSmall static sites with no build processA one-off snapshot that goes stale; free tiers often cap the number of URLs
Desktop crawler exportChecking what is linkable, then exportingIncludes only what it can reach through links
By hand in a text editorA handful of pagesTyping errors and unescaped ampersands

WordPress

WordPress has produced a basic sitemap at /wp-sitemap.xml since version 5.5. Yoast and Rank Math replace it with their own index at /sitemap_index.xml and let you exclude post types and taxonomies. See sitemap_index.xml in WordPress.

Next.js

// app/sitemap.ts
import type { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const products = await getProducts();
  return [
    { url: 'https://example.com/', lastModified: new Date('2026-10-01') },
    ...products.map((p) => ({ url: `https://example.com/products/${p.slug}`, lastModified: p.updatedAt })),
  ];
}

Online generators

A generator crawls your site from the home page and writes the file for you to download and upload to the root. It only finds pages reachable through links, and the file does not update itself, so regenerate it whenever you add or remove pages. For any site that changes, a CMS or build-time sitemap is the better long-term option.

How to check an XML sitemap before submitting

Check three things: the file is valid XML, every URL in it is a live canonical page, and the server delivers it correctly. Validate the syntax with xmllint, which is available on most Linux and macOS systems:

curl -s https://example.com/sitemap.xml -o sitemap.xml
xmllint --noout sitemap.xml && echo "well formed"

# list every URL and its status code
grep -o '<loc>[^<]*' sitemap.xml | sed 's/<loc>//' | while read u; do
  printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$u")" "$u"
done | sort | uniq -c -w3

Every line should start with 200. Anything showing 301, 404 or 5xx needs removing or fixing. Then open a sample of pages and confirm the canonical tag points at the same URL that is listed. The sitemap 404s guide and sitemap mismatches guide cover the usual problems.

๐Ÿ’ก Editing a sitemap by hand is fine for a small static site. Use a code editor that highlights XML, so a missing closing tag stands out, and run xmllint after each edit.

A worked example

Example Bakery submits /sitemap_index.xml from its SEO plugin and Search Console shows Success, but only 40 of 120 listed URLs are indexed. Running the status loop shows 30 URLs returning 301, because the plugin lists old http addresses from an imported product feed, and 25 tag archive pages that carry noindex. The bakery fixes the feed so products use https URLs, excludes tag archives in the plugin's sitemap settings and resubmits. The listed count drops to 65, all returning 200 and self-canonical, which gives Google a clean list to work through.

The fix, step by step

Step 1

Check whether you already have one

Look for a Sitemap: line in your robots.txt, then try /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. Check the Sitemaps report in Search Console for anything already submitted. Many platforms create a sitemap automatically, and running two in parallel, such as WordPress core and a plugin, only causes confusion. If one exists and is generated by your platform, skip to the checking steps rather than creating another.

Step 2

Choose how it will be generated

Use the built-in or plugin sitemap if you run WordPress, Shopify or another CMS, framework code such as app/sitemap.ts for a custom build, or an online generator or hand-written file for a small static site. Prefer a method that updates automatically when pages change. Write down which tool owns the sitemap, so nobody later adds a second generator that publishes a conflicting file.

Step 3

Decide which URLs belong in it

Include only pages you want in search results: canonical, indexable URLs that return 200. Exclude redirects, 404s, pages marked noindex, login and cart pages, internal search results and filtered or sorted duplicates. In a plugin, switch off post types and archives you do not want indexed. On a large site, split the list by type and plan a sitemap index that points at each file.

Step 4

Create the file with valid markup

Generate or write the file using the urlset element with the sitemaps.org namespace, one url element per page and a full https address in each loc. Add lastmod only where you know the real date of the last meaningful change. Escape any ampersand in a URL as the XML amp entity. Save as UTF-8 and place the file at the root, for example /sitemap.xml, unless your platform serves it for you.

Step 5

Validate the XML

Download the live file with curl and run xmllint --noout sitemap.xml. No output means it is well formed; an error message gives the line number of the problem. Also confirm with curl -sI that the address returns 200 with an XML content type rather than an HTML page. A file that opens in a browser can still fail validation, so do not skip this step.

Step 6

Check every listed URL

Extract each loc address and fetch its status code with a short curl loop, then count the results. Every URL should return 200. Remove or correct any that redirect or fail, and fix the generator setting that let them in so they do not come back. Spot-check that listed pages carry a canonical tag pointing to themselves and no noindex tag.

Step 7

Declare it in robots.txt

Add a line such as Sitemap: https://example.com/sitemap.xml to your robots.txt, using the full address. For a sitemap index, declare the index rather than each child file. Fetch robots.txt afterwards to confirm the line appears in the live file. This lets Bing and other search engines find the sitemap without needing it submitted to their own webmaster tools, and it keeps working if someone changes the Search Console setup.

Step 8

Submit it in Search Console

Open Sitemaps in Search Console for the correct property, enter the sitemap address and submit. The status should change to Success once Google has read it, and the report shows how many URLs it discovered. If it shows errors or Couldn't fetch, test the address again with curl. Over the following weeks, compare the listed pages with the Pages report to see how many are indexed.

Edge cases worth knowing

Keeping the sitemap accurate

Prefer a sitemap that is generated from your content, so it changes when pages are added or removed. Declare it in robots.txt as covered in creating a robots.txt file, keep it submitted in Search Console, and glance at the Sitemaps and Pages reports monthly. After a migration or redesign, rerun the status check, because old URLs often linger in the file.

Questions and answers

1. How do I create an XML sitemap for my website?

Most sites should let their platform create it: WordPress core, Yoast or Rank Math, Shopify and most website builders generate a sitemap automatically. For a custom site, generate it from your routes with framework code, or use an online generator or a hand-written file for a small static site. Whichever you choose, list only indexable pages that return 200, validate the file, add a Sitemap line to robots.txt and submit it in Search Console.

2. How do I find the XML sitemap on a site?

Open the site's robots.txt file and look for a line starting with Sitemap, which gives the full address. If there is none, try common addresses such as sitemap.xml, sitemap_index.xml and wp-sitemap.xml at the root of the domain. For your own site, the Sitemaps report in Google Search Console lists every sitemap that has been submitted. Platforms such as Shopify always publish theirs at sitemap.xml on the main domain, so that is always worth trying first.

3. How do I check if my XML sitemap is valid?

Download it and run an XML validator such as xmllint, which reports any broken tags or encoding errors with a line number. Then check that the server returns it with a 200 status and an XML content type, and that each URL inside returns 200 rather than a redirect or a 404. Finally, submit it in Search Console and confirm the status shows Success with the number of discovered URLs you expect.

4. Can I use an online XML sitemap generator?

Yes, for a small site that rarely changes. An online generator crawls your pages from the home page and gives you a file to upload to your site's root folder. The drawbacks are that it only finds pages reachable through links, free versions often limit the number of URLs, and the file becomes out of date as soon as you add or remove pages. A sitemap your CMS or build process generates is easier to keep accurate.

5. Can I edit my sitemap.xml by hand?

You can edit a static sitemap in any plain text or code editor, as long as you keep the XML structure intact and escape special characters such as ampersands. Validate the file after every change. If your sitemap is generated by a CMS or plugin, do not edit the output directly, because the next update will overwrite it; change the plugin settings or the content instead, such as excluding a post type or setting a page to noindex.

6. Do priority and changefreq matter in a sitemap?

Not for Google, which has said it ignores both values. They are part of the sitemaps.org protocol and are harmless to include, but there is little point maintaining them. The optional field that can matter is lastmod, which Google uses when it is consistently accurate. Set it to the date the page content genuinely last changed, not the date the file was generated, or Google will learn to disregard it.

7. Does a sitemap get my pages indexed?

No, a sitemap helps Google discover and recrawl URLs, but it does not guarantee indexing. Google still assesses each page on quality, duplication and whether it is allowed to be crawled and indexed. A sitemap is most useful for new sites, large sites and pages with few internal links. If submitted pages are not indexed, look at the Pages report in Search Console for the reason given for each URL.

Check every URL in your sitemap

The robots and sitemap checker reads your sitemap from robots.txt, validates it and tests the listed URLs for redirects, errors and noindex tags so you submit a clean file.

Open Robots and Sitemap Fixes โ†’
๐Ÿ’ฌ Got a problem?
Related Guides: All Robots and Sitemap Fixes  ·  Sitemap Could Not Be Read  ·  Sitemap Declaration in robots.txt  ·  Sitemap Index in WordPress  ·  Create a robots.txt File  ·  Learning Hub

About aiwebpageseo

aiwebpageseo.com is a data-driven SEO and AEO (Answer Engine Optimisation) platform providing a free suite of technical SEO tools. Rather than relying on AI-theorised assumptions, the platform analyses live URL performance, delivering objective diagnostics, page speed metrics, CLS debugging, and site crawl data alongside actionable technical tutorials.