How to edit robots.txt in WordPress, Yoast, Rank Math and Cloudflare

Where a WordPress robots.txt comes from

WordPress does not ship with a robots.txt file. When a request for /robots.txt reaches WordPress and no real file exists, WordPress builds one on the fly. This is the virtual robots.txt. Plugins can change the virtual file through a filter, and a CDN can change the response again on its way to the visitor. The order of precedence is what matters:

The default virtual file in current WordPress looks like this, with the Sitemap line pointing at the core sitemap unless an SEO plugin replaces it with its own:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

How to find out what is serving your file

Open https://www.example.com/robots.txt in a private window, then check from the command line so caches in your browser do not mislead you:

curl -s https://www.example.com/robots.txt
curl -sI https://www.example.com/robots.txt | grep -iE "^(server|cf-cache-status|age|last-modified|x-powered-by)"

Clues in the headers:

• last-modified usually appears when the web server sends a real file, and often not when PHP generates it.
• x-powered-by: PHP or WordPress-style headers suggest the virtual file.
• cf-cache-status: HIT with a large age means Cloudflare is serving a cached copy, which may be older than your latest edit.

If you have shell access, the definitive check is whether the file exists in the root:

ls -l /var/www/vhosts/example.com/httpdocs/robots.txt

That path is the Plesk default; on other hosts it is the folder that contains wp-config.php. Compare the file's contents with what the site serves. If they match, the physical file is in charge. If they differ, something between the file and the visitor is changing it, most often a CDN or a cache. If you see rules you did not expect, the robots.txt blocks guide explains how to tell which ones are doing harm.

Editing it in Yoast SEO and Rank Math

Yoast SEO

Go to Yoast SEO, then Tools, then File editor. If no robots.txt exists, Yoast offers a button to create one; it writes a real file to the site root, which then overrides the virtual file. Edit the text and save. If the File editor option is missing, file editing has been switched off on the site, usually by define( 'DISALLOW_FILE_EDIT', true ); in wp-config.php or by your host, or the folder is not writable. In that case upload the file by SFTP instead. Leave the protection in place; it exists for good security reasons.

Rank Math

Rank Math's editor is under Rank Math, then General Settings, then Edit robots.txt, and the menu item may only show in Advanced Mode. Rank Math does not write a file. It stores your text and serves it as the virtual robots.txt. That has one consequence that catches many people: if a physical robots.txt exists, the box is greyed out and Rank Math shows a notice. Rename or delete the physical file by SFTP and the editor unlocks.

Using a code filter instead

Without an SEO plugin, you can add to the virtual file from a small plugin or your child theme's functions.php:

add_filter( 'robots_txt', function ( $output, $public ) {
    $output .= "Disallow: /?s=\n";
    $output .= "Disallow: /search/\n";
    return $output;
}, 20, 2 );

This only affects the virtual file, so it does nothing if a physical file exists. Use one method only. A physical file, a plugin and a filter all trying to set the same file is the usual reason for confusion. For the wider WordPress picture, including sitemaps, see robots.txt and sitemaps in WordPress.

Editing the file directly

Without a plugin, create a plain text file named robots.txt, all lower case, in the site root. Use a text editor rather than a word processor so no formatting is added. Save as UTF-8, with one rule per line. Upload it by SFTP or through your hosting file manager; in Plesk that is Files, then the httpdocs folder for the main domain.

On the server, editing in place avoids a download and re-upload:

sudo nano /var/www/vhosts/example.com/httpdocs/robots.txt
sudo chown siteuser:psacln /var/www/vhosts/example.com/httpdocs/robots.txt
sudo chmod 644 /var/www/vhosts/example.com/httpdocs/robots.txt

Replace siteuser with the subscription's system user. The file must be readable by the web server, or it returns 403, which Google treats as no rules. On nginx, check that /robots.txt is allowed to fall through to WordPress when no file exists. Some configurations serve it only as a static file, so the virtual file returns 404 and plugin edits never appear:

location = /robots.txt {
    try_files $uri $uri/ /index.php?$args;
    access_log off;
}

How Cloudflare can change robots.txt

Cloudflare can do two things to the file. The first is caching: plain text files are among the types Cloudflare caches by default, so after an edit the edge may keep serving the old copy until it expires or you purge it. In the dashboard, go to Caching, then Configuration, and use Custom Purge to purge the exact address https://www.example.com/robots.txt, plus the non-www version if it serves its own file.

The second is adding content. Cloudflare offers a managed robots.txt option, found with its bot and AI crawler settings, which adds Cloudflare's own lines to your file, such as rules telling known AI training crawlers not to crawl. If your live file contains lines you never wrote, especially near the top, check that setting before searching WordPress for them. Turn it off if you prefer to manage every line yourself, or keep it and write your own rules around it.

Remember that robots.txt is a request, not a lock. Well-behaved crawlers follow it; others ignore it. To actually stop unwanted bots, use firewall rules, and to decide which AI crawlers to allow, see fixing robots.txt AI blocks.

A sensible WordPress robots.txt

Most WordPress sites need very little. This is a good starting point:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

The search lines stop crawlers wasting time on internal search results, which can produce endless thin URLs. Use the sitemap address your setup actually serves: /sitemap_index.xml for Yoast and Rank Math, /wp-sitemap.xml for WordPress core.

Never use robots.txt to remove a page from Google. It stops crawling, not indexing, and a blocked page can still appear in results with no description. To keep a page out, leave it crawlable and add noindex, as the noindex and disallow guide explains.

A worked example

An owner manages robots.txt in Rank Math. After restoring the site from a backup, they want to add Disallow: /?s=, but the live file no longer matches anything in Rank Math, and the editor box is now greyed out. Clearing the browser cache and purging the caching plugin make no difference.

Listing the site root by SFTP shows an old robots.txt written years earlier by a plugin that has since been removed, brought back by the backup. The web server serves that file before WordPress ever runs, so Rank Math's virtual version never reaches anyone. The owner renames the old file to robots-old.txt, purges /robots.txt in Cloudflare, and reloads. Rank Math's version appears at once, including the new rule.

The lesson is to check the site root after any restore or migration. A stray physical file is invisible from the WordPress dashboard, yet it decides what every crawler reads.

The fix, step by step

Edge cases worth knowing

• Multisite. All sites in a subdirectory network share one root, so one physical robots.txt applies to every site. A virtual file can differ per site.
• www and non-www. If both hosts serve content, each needs its own correct file. Usually one redirects to the other, and the file at the final address is the one that counts.
• Staging sites. A staging robots.txt with Disallow: / copied to live is a common launch mistake. Check the file on launch day.
• Search engine visibility. The Discourage search engines box under Settings, then Reading, adds a noindex to pages. It should be unticked on a live site.
• Caching plugins. Some page caches store /robots.txt like any page. Purge it, or exclude it from caching, after each change.

Keeping it fixed

Pick one source and stick with it: a plugin's virtual file or a physical file, not both. Write down which one you use somewhere your team will find it. After every change, load the live file with curl, purge the CDN copy, and check the robots.txt report under Settings in Search Console, which shows when Google last fetched the file and whether the fetch worked. If the file ever fails to load, the robots.txt unreachable guide explains why that is urgent.