How to Fix Robots.txt and Sitemap in Next.js
Next.js 13+ App Router introduces app/robots.ts and app/sitemap.ts — TypeScript-based dynamic generation of these files. Cleaner than maintaining static files. This guide covers Next.js-specific robots and sitemap. Pair with robots/sitemap guide.
Step-by-step: How to fix robots.txt and sitemap in Next.js
- Create app/robots.ts (App Router). Export default function returning MetadataRoute.Robots: { rules: [{ userAgent: '*', allow: '/', disallow: ['/admin', '/api'] }], sitemap: 'https://example.com/sitemap.xml' }. Next.js generates /robots.txt at build.
- Add AI bot policy. Multiple rule blocks per user-agent. For example: rules: [{ userAgent: 'GPTBot', allow: '/' }, { userAgent: 'PerplexityBot', allow: '/' }, { userAgent: 'CCBot', disallow: '/' }, { userAgent: '*', allow: '/', disallow: ['/admin'] }]. Document AI policy.
- Create app/sitemap.ts. Export default async function returning MetadataRoute.Sitemap: [{ url: 'https://example.com', lastModified: new Date(), changeFrequency: 'weekly', priority: 1 }, ...]. Fetch your data, map to sitemap entries, return.
- Generate sitemap dynamically from data. For blogs: fetch all posts → map to { url, lastModified, ... } entries. For products: same with product data. Append static URLs (homepage, about, etc.). Return combined array.
- For Pages Router: use next-sitemap. npm install --save-dev next-sitemap. Configure next-sitemap.config.js. Runs as postbuild script generating public/sitemap.xml. Pages Router doesn't have built-in sitemap.ts.
- Handle large sitemaps. Single sitemap.xml has 50,000 URL limit. For larger sites, use sitemap index: app/sitemap.ts returns multiple sitemap files (sitemap-products.xml, sitemap-blog.xml, etc.). next-sitemap handles this with siteUrl + sitemapSize config.
- Submit to Search Console. Search Console → Sitemaps → add sitemap.xml URL. Verify it processes successfully. Repeat in Bing Webmaster Tools.
🤖 Audit Next.js robots/sitemap
Validate Next.js robots and sitemap configuration.
Run Robots/Sitemap Audit →Frequently Asked Questions
App Router vs Pages Router for robots/sitemap?
App Router: built-in app/robots.ts and app/sitemap.ts — TypeScript, type-safe, dynamic generation. Pages Router: no built-in support — use next-sitemap library or static public/robots.txt file. New projects should use App Router pattern; existing Pages Router projects continue with next-sitemap.
Should robots.ts be dynamic or static?
Either works. Dynamic (robots.ts as TypeScript function) is useful when rules differ by environment (dev: noindex all, prod: index). Static (public/robots.txt file) is fine for stable rules. Use dynamic when you want env-conditional logic.
How do I handle locale-aware sitemaps in Next.js?
sitemap.ts returns entries with alternates: { languages: { en: '...', fr: '...' } } property. Generated sitemap.xml includes hreflang information per URL. For multi-locale sites this is essential.
Best Next.js sitemap library?
Built-in App Router app/sitemap.ts for new projects (no library). next-sitemap for Pages Router or projects needing more features (sitemap index, custom transforms, exclude patterns).
Why isn't my Next.js sitemap appearing at /sitemap.xml?
If using app/sitemap.ts, ensure file exports default function. Build the project (npm run build). Verify .next/server/app/sitemap.xml exists. Test by visiting /sitemap.xml on running server. Common error: file named sitemap.tsx (page) instead of sitemap.ts (config).