Server Error (5xx) in Search Console: How to Find and Fix It

What Server error (5xx) actually means

Every response a web server sends starts with a three-digit status code. Codes from 500 to 599 mean the server accepted the request but failed to complete it. When Googlebot receives one of these for a URL, or the connection times out before any answer arrives, Search Console lists that URL under Server error (5xx) in the Page indexing report.

The important word is when. Search Console records what happened on Google's last crawl of that URL. If your server was overloaded for twenty minutes at 3 am, every URL Google happened to request in that window can be listed, even though they all work now. That is why the first job is never to stare at the page in your browser; it is to find out what the server was doing at the time Google called.

Google treats 5xx codes, and also 429 Too Many Requests, as a sign that the server is struggling. Its crawlers respond by slowing down. If the errors continue, URLs that were indexed can be removed, and new URLs wait longer to be crawled at all. A handful of 5xx URLs after a known outage is not a crisis. A steady trickle that keeps growing is.

The usual causes and how to tell them apart

Most 5xx errors seen by Googlebot come from a short list of causes. The pattern in the report, and what the logs say at the time, tells you which one you have.

If only some URLs fail, look for what they share: a slow template, a heavy query or a plugin that runs only on those pages.

How to find the failed requests in your logs

Start in Search Console. Open the Server error (5xx) reason in the Page indexing report and pick three or four example URLs. Run each through URL Inspection and note the Last crawl date and time. Now you have moments to look for.

Next, check that the URLs work right now from outside your network:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" https://example.com/some-page/
curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/some-page/

If the second command returns 403, 429 or 503 while the first returns 200, a security rule is treating the Googlebot user agent differently. If both return 200 quickly, the problem was temporary and the logs will tell you what happened.

Search the access log

List Googlebot requests that ended in a 5xx code, with the time and URL. The log location depends on the host; on Plesk it is usually under /var/www/vhosts/system/example.com/logs/, and on a plain Apache or nginx server under /var/log/apache2/ or /var/log/nginx/.

grep "Googlebot" access_log | awk '$9 ~ /^5/ {print $4, $9, $7}' | sort | uniq -c | sort -rn | head -50

This counts each time, status and path combination, so clusters stand out. The field numbers assume the common combined log format; check one line and adjust if yours differs.

Read the error log for the same minutes

The access log tells you what failed. The error log tells you why. Open the error log and look at the minutes around each failure:

grep -i -E "fatal|max_children|too many connections|upstream timed out|out of memory" error_log | tail -40

Typical lines and what they mean:

• PHP Fatal error: Allowed memory size exhausted: a page needs more memory than PHP allows, or a plugin is leaking.
• server reached pm.max_children setting: PHP-FPM has no free workers, so requests queue and then fail.
• Too many connections: the database has hit its connection limit.
• upstream timed out or connect() failed in the nginx proxy log: the application behind nginx was too slow or not running.

Check the request really came from Google

Plenty of scrapers claim to be Googlebot. Before you change any rules, confirm an IP address from the log with a reverse lookup and then a forward lookup:

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

A genuine Googlebot address resolves to a host name ending in googlebot.com, google.com or googleusercontent.com, and that host name resolves back to the same IP. Google also publishes its crawler IP ranges as JSON files, which most firewalls can import.

If you would rather not work through logs by hand, the HTTP error audit and the log analyser show which URLs return errors and how often crawlers hit them.

The key changes that stop the errors

Capacity: workers, memory and database

If errors cluster at busy times, the server is running out of something. The quickest gains usually come from caching, because a cached page costs almost nothing to serve. A full-page cache in front of WordPress or another CMS can take most crawler traffic off PHP and the database entirely. After that, raise the limits that the error log names, within what the server's memory can support:

; PHP-FPM pool, for example /etc/php/8.3/fpm/pool.d/www.conf
pm = dynamic
pm.max_children = 20
pm.start_servers = 4
pm.min_spare_servers = 2
pm.max_spare_servers = 6

# MySQL or MariaDB, my.cnf
[mysqld]
max_connections = 200

Size pm.max_children from memory: at about 80 MB per PHP process and 1.6 GB to spare, 20 is the ceiling. Going higher swaps 503s for a server that grinds to a halt.

Timeouts on proxies and CDNs

A 502 or 504 from nginx in front of PHP-FPM or Node.js often means the proxy gave up first. Raise the proxy timeout only after you have looked at why the page is slow, but make sure it is not shorter than your slowest legitimate page:

location ~ \.php$ {
    fastcgi_read_timeout 60s;
}
location / {
    proxy_connect_timeout 10s;
    proxy_read_timeout 60s;
}

Pages that take tens of seconds to build are a problem in their own right. See fixing 504 errors and fixing slow templates.

Planned downtime: use 503 with Retry-After

During maintenance, return 503 with a Retry-After header, never a 200 page saying the site is down and never a redirect to a holding page. Google treats a short period of 503 as temporary and comes back later.

# nginx: maintenance mode for everyone except your own IP
location / {
    if ($remote_addr != 203.0.113.10) {
        add_header Retry-After 3600 always;
        return 503;
    }
}

Platform notes

• WordPress. Turn on WP_DEBUG_LOG temporarily to catch fatal errors in wp-content/debug.log, and check the security plugin's firewall log for blocked Googlebot requests. WP-Cron runs scheduled tasks on page loads by default, which can make heavy jobs run during crawls; move them to a real cron job. See HTTP errors in WordPress.
• Plesk. Each domain's logs are under Websites and Domains, then Logs, or on disk in /var/www/vhosts/system/example.com/logs/. With nginx in front of Apache, check proxy_error_log as well as error_log. PHP-FPM limits are set per domain in PHP Settings. See HTTP errors in Plesk.
• Cloudflare. Errors 520 to 526 come from Cloudflare and mean the origin did not answer properly. Error 524 means the origin took longer than Cloudflare's timeout, which is 100 seconds on most plans. Check Security Events for challenges served to verified bots; Cloudflare's verified bot list normally lets Googlebot through, but custom rules can still block it.
• Node.js and Next.js. A crashed process behind a proxy produces 502s until it restarts. Use a process manager that restarts on failure, log unhandled errors, and watch for memory growth. See HTTP errors in Node.js.

A worked example

An online shop sees 200 product pages under Server error (5xx). URL Inspection on five of them shows last crawl times between 02:05 and 02:40 on the same morning. Every page loads in under a second now.

Searching the access log for that half hour shows 340 Googlebot requests, 212 of them with status 503. The error log for the same minutes repeats one line: the database has reached its connection limit. The cron table shows a nightly stock import starting at 02:00 that opens a connection per product.

The fix has two parts: the import moves to 04:30, after Google's usual busy period for that site, and is changed to reuse a single connection; max_connections is raised from 100 to 200. The next night the log shows no 503s. The owner chooses Validate fix in Search Console, and over the following fortnight the URLs move out of the error list as Google recrawls them.

The fix, step by step

Edge cases worth knowing

• Errors you cannot find in your logs. If your origin log shows a 200 but Google saw a 5xx, the error happened in front of your server, at a CDN, load balancer or security proxy. Check that layer's logs.
• Timeouts with no status code. Google may list a URL as a server error when the connection timed out and no status was ever sent. Your access log may have no line at all, or a line with an unusually long response time.
• Errors on robots.txt. If robots.txt itself returns 5xx, Google may pause crawling the whole site until it can read the file. That is a separate and more urgent problem; see robots.txt unreachable errors.
• 429 instead of 503. Rate limiting that answers 429 has the same effect on Googlebot as a 5xx. Exclude verified Googlebot addresses from rate limits rather than slowing them down with errors.
• URLs that should not exist. If the failing URLs are junk, such as filter combinations or old search pages, fixing the 5xx is only half the job. They should return 404 or be kept out of crawling, as covered in crawl budget fixes.

Keeping server errors away

Once the errors stop, keep watching for a few weeks. Check the Crawl stats report weekly: the share of server error responses should stay near zero, and the Host status panel should show no recent problems. A sudden rise usually follows a plugin update, a new scheduled job or a traffic spike.

• Set up an uptime monitor that requests a few key pages every minute and alerts you on any 5xx.
• Keep a page cache in place and check its hit rate after every theme or plugin change.
• Schedule backups, imports and feed generation for your quietest hours, and look at your own logs to find them.
• Rotate and keep logs for at least 30 days, so you can always look back to the time of a Google crawl.
• Test plugin and theme updates on a staging copy before they reach the live site.

Running a regular site audit alongside these checks catches new errors before Google reports them.