How to Create, Find and Read a robots.txt File, With Examples
A robots.txt file tells search engine crawlers which parts of your site they may request. Getting it right is quick, but getting it wrong can be severe: one stray line can stop Google crawling your whole site, while a missing rule can let crawlers waste their time on endless filter pages. Many site owners are also unsure where the file lives, or why it appears in the browser but not on the server. This guide shows how to find any site's robots.txt, how to read each rule, how to write a file from scratch with safe examples for common setups, how to create it on WordPress, Shopify, Next.js and ordinary hosting, and when Disallow all is and is not appropriate.
What robots.txt is and what it controls
A robots.txt file is a plain text file at the root of a host that tells crawlers which paths they may and may not request. Well-behaved crawlers such as Googlebot and Bingbot fetch it before crawling anything else on that host and follow its rules. It is a crawling instruction, not a security feature: anyone can read it, and badly behaved bots can ignore it.
Three facts shape everything else:
- It lives at one fixed address. Crawlers only look at
/robots.txton the exact host and protocol.https://example.com/robots.txtdoes not apply tohttps://shop.example.com, which needs its own file. A file in a subfolder is ignored. - It blocks crawling, not indexing. A disallowed URL can still appear in Google if other pages link to it, usually with no description. To keep a page out of results, use
noindexand let Google crawl it. - No file is fine. If
/robots.txtreturns 404, Google assumes it may crawl everything. You only need a file when you want to restrict something or list your sitemap.
Google reads up to 500 KiB of the file, expects UTF-8 plain text, and usually caches it for up to 24 hours, so changes are not seen instantly.
How to find a website's robots.txt file
To find any site's robots.txt, type the domain followed by /robots.txt into the address bar, for example https://example.com/robots.txt. If the browser shows plain text with lines such as User-agent, that is the file. If you get a 404 page, the site has none.
From a terminal you can also see the status code, which matters because a 5xx error is treated very differently from a 404:
curl -sI https://example.com/robots.txt
curl -s https://example.com/robots.txtFor your own site, Search Console has a robots.txt report under Settings. It lists the robots.txt files Google found for your property, when each was last fetched, the status, and any warnings or errors in the rules. You can request a recrawl of the file from there after you change it.
Where the file is on your server
On a typical Apache, nginx or Plesk host the file sits in the document root, the same folder as your home page, often httpdocs or public_html. On WordPress, Shopify and some frameworks there may be no physical file at all, because the platform generates it on request. That is why you sometimes see a robots.txt in the browser but cannot find it over FTP.
How to read robots.txt rules
A file is made of groups. Each group starts with one or more User-agent lines naming the crawler, followed by Disallow and Allow rules. A crawler obeys only the most specific group that matches its name, and falls back to the * group if none does.
| Line | What it means |
|---|---|
User-agent: * | The rules below apply to every crawler without its own group |
User-agent: Googlebot | The rules below apply to Google's main crawler only |
Disallow: /admin/ | Do not request any URL whose path starts with /admin/ |
Disallow: (empty) | Nothing is blocked |
Disallow: / | Everything on the host is blocked |
Allow: /admin/help.html | Exception to a broader Disallow |
Disallow: /*?sort= | * matches any characters, so any URL containing ?sort= is blocked |
Disallow: /*.pdf$ | $ marks the end of the URL, so only paths ending in .pdf |
Sitemap: https://example.com/sitemap.xml | Location of a sitemap; must be a full URL; applies to the whole file |
Paths are case-sensitive and match from the start. When an Allow and a Disallow both match, Google applies the rule with the longest path, and if they are equally long, the Allow wins. Google ignores Crawl-delay and has not supported Noindex lines in robots.txt since 2019, so neither does anything for Google. The robots.txt syntax guide covers the mistakes that make rules silently fail.
Reading a real file, line by line
User-agent: *
Disallow: /checkout/
Disallow: /*?filter=
Allow: /checkout/help
User-agent: Googlebot-Image
Disallow: /private-images/
Sitemap: https://example.com/sitemap.xmlMost crawlers use the first group: they skip the checkout and any URL containing ?filter=, but may fetch /checkout/help because that Allow rule is longer than the Disallow it overrides. Google's image crawler has its own group, so it follows only that group and ignores the first one entirely. That catches people out: once a crawler has a named group, rules in the * group no longer apply to it, so repeat any rules it also needs. The Sitemap line sits outside the groups and applies to everyone.
robots.txt examples for common setups
Allow everything and list the sitemap
This is the right file for most small business sites:
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlBlock a few private or low-value areas
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?sort=
Disallow: /search
Sitemap: https://example.com/sitemap.xmlWordPress default with admin AJAX open
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlDisallow all, for a staging copy
User-agent: *
Disallow: /Never block the CSS, JavaScript or image folders your pages need, because Google renders pages and a blocked stylesheet can make a page look broken to it. Choosing whether to block AI crawlers is a separate decision covered in robots.txt and AI crawlers.
Creating the file on your platform
- Static, Apache, nginx or Plesk. Create a file called
robots.txt, all lower case, in the document root. In Plesk use File Manager inhttpdocs; on a server you can write it in place withsudo nano /var/www/example.com/robots.txt. Save it as UTF-8 plain text. - WordPress. WordPress serves a virtual file if no real one exists. SEO plugins such as Yoast and Rank Math have a robots.txt editor; alternatively, upload a physical file to the root, which overrides the virtual one. See robots.txt and sitemaps in WordPress.
- Shopify. Shopify generates a sensible default. To change it, add a
robots.txt.liquidtemplate in the theme code editor. See robots.txt in Shopify. - Next.js. In the App Router, add
app/robots.tsexporting the rules, or place a staticrobots.txtinpublic. See robots.txt in Next.js. - Cloudflare. Cloudflare can add its own managed lines to your file if you enable certain bot settings, so check the live output, not just your source file.
A worked example
Example Bakery launches a new site and finds that no new pages appear in Google after two weeks. Fetching the file shows:
$ curl -s https://example.com/robots.txt
User-agent: *
Disallow: /The developer's staging file went live with the site. The Search Console robots.txt report shows the same content fetched the day after launch. The bakery replaces it with the allow-everything file plus its sitemap line, then uses the report's option to request a recrawl of robots.txt. Within a day the report shows the new version, and URL Inspection on the home page no longer reports that crawling is blocked. Pages begin appearing over the following days as Google crawls them.
The fix, step by step
Step 1
Check whether a file already exists
Open https://yourdomain/robots.txt in a browser and run curl -sI on the same address to see the status code. A 200 with text means a file exists, a 404 means there is none, and a 5xx means your server is failing on it, which must be fixed first. Repeat for every subdomain you use, because each host needs its own file. Note whether your platform generates the file for you.
Step 2
Decide what genuinely needs blocking
List areas that crawlers should not spend time on: admin screens, carts, checkouts, account pages, internal search results and parameter variants such as sort orders. Leave out anything you want in Google, and never block CSS, JavaScript or image folders that pages need to render. If a page must stay out of search results, plan to use noindex on it instead of a Disallow rule, because blocked pages can still be indexed.
Step 3
Write the file in a plain text editor
Create a UTF-8 plain text file named robots.txt in lower case. Start with User-agent: *, add one Disallow line per path to block, and add Allow lines only for exceptions. Finish with a Sitemap line giving the full https address of your sitemap. Keep it short. A small business site often needs no more than four or five lines in total.
Step 4
Put it at the root of the host
Upload the file to the document root, the folder that holds your home page, using File Manager, SFTP or a server command such as sudo nano. On WordPress use your SEO plugin's editor or upload a real file, on Shopify edit robots.txt.liquid, and in Next.js add app/robots.ts. The file must be reachable at /robots.txt exactly, not in a subfolder, and the name must be all lower case.
Step 5
Confirm the live output
Fetch the file again with curl -s https://yourdomain/robots.txt and read it line by line. Make sure the content is what you wrote, that a CDN, plugin or platform has not added or changed lines, and that the status is 200 with a text content type. If you see old content, purge the cache on your CDN or caching plugin and fetch it again.
Step 6
Test important URLs against the rules
Pick your home page, a key service page, a product page and one URL you meant to block. Work through the rules for each, remembering that the longest matching path wins and an Allow wins a tie. Then use URL Inspection in Search Console on the pages you want indexed and check that it does not report the page as blocked by robots.txt.
Step 7
Ask Google to fetch the new version
Open the robots.txt report under Settings in Search Console. Check that the file is listed with a recent fetch and no errors, then use the option to request a recrawl so Google picks up the change sooner than its normal cache period of up to a day. Check the report again the next day and confirm the fetched version matches your live file.
Edge cases worth knowing
- Server errors. If robots.txt returns a 5xx error, Google stops crawling the site for a while, treating it as fully disallowed. A 404 is harmless; a 500 is not. See robots.txt unreachable errors.
- Redirects. Google follows a few redirects to fetch robots.txt, so http to https is fine, but a redirect to another domain applies that domain's rules.
- Blocking a page you also noindex. If a page is disallowed, Google cannot see its
noindextag, so it may stay indexed. The noindex and disallow guide explains the order to do things in. - Byte order marks and file type. A file saved as rich text or with odd encoding can break the first line. Use a plain text editor.
Keeping robots.txt right
Treat robots.txt as part of your deployment. Keep it in version control, use a different file for staging that is never copied to live, and check the live file with curl after every launch or migration. Look at the Search Console robots.txt report every few months for new warnings. When you add a new section of the site, ask whether it needs a rule, and keep your sitemap declaration pointing at a sitemap that actually exists.
1. How do I find the robots.txt file of a website?
Add /robots.txt to the end of the domain, for example example.com/robots.txt, and open it in a browser. The file always lives at the root of the host, so there is nowhere else to look. If you see a 404 page, the site has no robots.txt and crawlers may access everything. For your own site, the robots.txt report under Settings in Google Search Console shows the version Google last fetched and any problems it found.
2. How do I make a robots.txt file?
Open a plain text editor, write a User-agent line followed by any Disallow rules and a Sitemap line, and save the file as robots.txt in UTF-8. Upload it to the root folder of your site so it loads at yourdomain/robots.txt. On WordPress, Shopify and Next.js you usually create or edit it through the platform instead, because they generate the file. Afterwards, fetch the live address to confirm it shows exactly what you wrote.
3. What does Disallow all do in robots.txt?
A group of User-agent: * with Disallow: / tells every compliant crawler not to request any URL on the host. Search engines stop crawling the site, so they cannot refresh its pages, and over time it drops out of results, although some URLs may remain listed without descriptions. It is meant for staging or private copies of a site, and even there a password is safer. On a live business site it is almost always a mistake.
4. Does robots.txt stop a page being indexed?
No. Robots.txt only stops crawling. If other pages link to a blocked URL, Google can still index the address and show it in results, usually without a description because it has not read the page. To keep a page out of the index, allow it to be crawled and add a noindex robots meta tag or an X-Robots-Tag header, so Google can see the instruction. Use robots.txt to save crawl effort, not to hide pages.
5. Do I need a robots.txt file at all?
Not strictly. If the file is missing and returns a 404, Google treats the whole site as crawlable, which is fine for many small sites. A robots.txt file becomes useful when you want to keep crawlers out of areas such as carts, account pages or endless filter combinations, or to point them at your sitemap. What you must avoid is a file that returns a server error, because Google then pauses crawling.
6. How do I read the rules in a robots.txt file?
Read it group by group. Each group starts with User-agent lines saying which crawler it is for, and the Disallow and Allow lines below say which paths that crawler may not or may request, matched from the start of the path. A star matches any characters and a dollar sign marks the end of a URL. A crawler uses the most specific group that names it, and within a group the longest matching rule decides.
7. How long does Google take to notice a robots.txt change?
Google generally caches robots.txt for up to 24 hours, so a change is usually picked up within a day. You can speed this up by requesting a recrawl from the robots.txt report in Search Console. Bear in mind that noticing the new rules is not the same as recrawling the pages they affect: pages that were blocked will be crawled again gradually, and changes in search results can take days or weeks to follow.
The robots and sitemap checker fetches your live robots.txt, reads every rule, tests your key pages against it and confirms that the sitemap it declares can be read.
Open Robots and Sitemap Fixes →