How to Fix Every Real Traffic Audit Finding

Which finding to fix first

The audit is a measurement tool, so most of what it reports is information rather than a fault. A finding only needs work when it means one of three things: visitors are being let down, the site is exposed, or the numbers you use to make decisions are wrong. Rank anything you see by that order.

Jump straight to the finding you are looking at:

• Your own server inflating analytics: requests from your server IP, which can inflate analytics when they load pages in a browser.
• Bots and automated browsers in analytics: monitors, speed testers and script-running bots counted as new users.
• AI crawler access: training crawlers, AI search crawlers and live assistant fetches.
• Errors real visitors hit: error responses served to real visitors rather than bots.
• Security events: failed logins, password-guessing bursts, admin access, new accounts, probes and error bursts.
• Short, partial or unreadable logs: retention, CDN caching and log format.
• Analytics and Search Console set-up: shared Measurement IDs, consent and verification.

Your own server inflating analytics

The section Your Own Server & Automated Browsers counts requests that came from the server IP address you typed into the optional field. Those requests are your website talking to itself. A few are normal: a scheduled task calling a script, a cache being warmed, a health check. They become a problem for analytics when the server loads pages in a real browser engine, because the page then runs your tracking tag and each load can be counted as a brand-new user that never keeps a cookie. The tool page describes one site where half of all page loads came from the site itself.

Exclude the server in Google Analytics

1. In GA4 open Admin → Data streams, choose your web stream, then Configure tag settings → Show more → Define internal traffic.
2. Create a rule with the traffic type value internal and add your server's public IP address. Add your office and home addresses too.
3. Go to Admin → Data collection and modification → Data filters, open the Internal Traffic filter and set it to Active.

This cannot hide genuine visitors, because nobody browses your site from your server's own address. Bear in mind that GA4 data filters apply from the moment they are switched on and do not clean up figures already collected, so expect older reports to stay inflated. If you want to check the rule before committing, a filter can be left in its testing state for a few days first.

Then find out why the server loads its own pages

Excluding the address treats the symptom. It is worth asking what is doing the loading, because it also uses CPU and memory you are paying for. Common causes are a headless browser used to generate PDFs, social images or screenshots of your pages; a pre-rendering or server-side rendering set-up that fetches full pages; a cache-warming script that requests every URL on a schedule; and plugins that call the site's own front end through its public address. Look at what the report shows for your server IP, then check your scheduler (cron, Plesk Scheduled Tasks, cPanel Cron Jobs, or your CMS's task list) for anything that runs at matching times.

Bots and automated browsers in analytics

Google Analytics already drops traffic from bots and spiders on industry known-bot lists, and you cannot switch that off. What slips through is software that behaves like a browser: uptime and speed-testing services that use a real browser engine, audit tools, scrapers that execute JavaScript, and headless browsers driven by scripts. The audit separates the ones that identify themselves, under headless browsers and tools or other bots and scanners. If your analytics user count is far higher than the likely people figure for the same dates, these are usually part of the gap.

Ways to stop them being counted

• Your own monitors. Point uptime checks at a lightweight URL that carries no analytics tag, such as a plain health-check file, rather than your home page. Many monitoring services publish their source IP ranges; add those to an internal traffic rule if they must load real pages.
• Speed and audit tools you run. Run them less often, or add their address to an internal traffic rule. A daily scheduled test of ten pages adds up to thousands of phantom users a year.
• Automation you control. Browsers driven by automation frameworks normally report navigator.webdriver as true. A small condition in your tag loading code can skip analytics when that flag is set. It does nothing about bots that hide it, but it removes your own tooling cleanly.
• Bots you do not control. These cannot be fixed inside analytics. Use the report to judge their scale, and if a scraper is heavy enough to cost you server resources, rate-limit or block it at the firewall, CDN or web server.

Expect some difference to remain. Analytics counts devices and cookies; the audit counts IP addresses. Several people in one office share an address, and one person moving between Wi-Fi and mobile data gets two. Aim to explain the gap rather than make it zero.

Deciding what AI crawlers may fetch

The AI Crawlers & AI Assistants section splits AI traffic into three kinds, and each calls for a different decision. Training crawlers such as GPTBot and ClaudeBot collect pages for future models. AI search crawlers such as OAI-SearchBot and PerplexityBot build the indexes assistants cite from today. Live fetches such as ChatGPT-User, Claude-User and Perplexity-User happen because a person asked an assistant about your page. When the same live fetch arrives at the same minute on several days, the audit flags it as scheduled, because people do not ask at an identical time every day.

A sensible default for most businesses

Allow the AI search crawlers and live fetchers if you want to be mentioned and linked in AI answers. Decide separately about training crawlers: blocking them does not stop assistants finding you through search, but it does withhold your text from future model training. Publishers and anyone selling content often block training; most small service businesses gain more from being visible.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/

Remember that a crawler obeys only the group that names it most specifically, so a bot listed in its own group ignores the rules under User-agent: *. Repeat any shared disallow lines inside each named group. robots.txt is a request, not a lock: reputable crawlers follow it, others do not, and providers have said that fetches a person triggers directly may be treated differently from automated crawling. Check each provider's current documentation before relying on it, and block at the server or CDN if you need enforcement.

The AI crawler access guide covers user agent names in more depth, and fixing accidental AI blocks helps when the report shows crawlers you wanted are absent.

Fixing errors real visitors hit

Error lists from crawlers are long and mostly noise: bots request old URLs and invented paths endlessly. Errors Real Visitors Hit is narrower and more useful because it only shows failures served to browser-like visitors. Each line is a person who wanted something and did not get it.

Missing pages (404)

• The page moved. Add a permanent 301 redirect from the old address to the closest equivalent. Do not send everything to the home page; that frustrates people and search engines treat it much like a 404.
• Your own link is wrong. Search your site for the broken address and correct the link at source, then add a redirect for anyone who bookmarked it.
• Someone else links to a typo. Redirect the misspelt address to the right page; it costs nothing and rescues the visit.
• The content is gone for good. A 410 status says so clearly. Make sure the error page itself offers search and your main navigation.

Server errors (5xx)

A 500 means the application failed. A 502 means the web server got an invalid response, or none at all, from the back end such as PHP or Node; a 504 means it timed out waiting for one. Note the failing path, then check the server's error log around that time, which usually names the script, plugin or limit involved. Repeated errors on one form or checkout path deserve immediate attention because they often hit the visitors most likely to buy. The 404 guide, 500 guide and 502 guide go step by step, and the HTTP Error Audit fixes cover status codes across a whole site.

Acting on security events

Security Events reports failed logins, password-guessing bursts, admin areas reached, new accounts, attack probes and bursts of server errors. Attack probes on their own are the background noise of the internet: automated scans for WordPress logins, leaked .env files and similar weak spots reach every public site and are not evidence of a breach. The other events deserve a closer look.

If an unfamiliar address reached your admin area, treat it as a possible compromise: change every administrator password, rotate API keys and database credentials stored in configuration files, check for recently changed files, and ask your host for help if you are unsure. The report's results file for the Security Audit contains the security events, for use in the Security Audit. See also the Security Audit fixes.

Short, partial or unreadable logs

The audit can only describe what your log contains. Three set-up problems quietly limit it.

Too little history

Many hosts keep only days or a couple of weeks. In Plesk, raise the number of files kept under Websites & Domains → Logs → Manage Log Rotation. In cPanel, open Raw Access and tick the option to archive log files to your home directory, and untick the one that removes the previous month's archive. On your own server, logrotate controls it: in /etc/logrotate.d/nginx or the Apache equivalent, the rotate value sets how many old files are kept. Weigh history against privacy, because access logs hold IP addresses, which are personal data. Choose a period you can justify, state it in your privacy notice, and delete downloaded copies when you have finished. This is general information, not legal advice.

Traffic the CDN answers

Pages served entirely from a CDN cache never reach your server, so they are missing from the log. Many CDNs cache images, styles and scripts by default but pass HTML through, in which case page views still appear but the supporting files may not. Since the audit's likely-people test looks for a browser loading a page together with its images, styles or scripts, heavy asset caching can make real visitors harder to tell apart from bots. If you can, analyse a period when caching settings were stable, and keep in mind that the log describes what your server handled rather than every visit.

Unrecognised lines

The audit reads the standard combined format used by Apache, nginx, Plesk and cPanel; anything else is counted as unrecognised. In nginx the built-in combined format is the default for access_log, so a custom log_format is the usual culprit. In Apache, make sure the CustomLog line ends in combined rather than common, which leaves out the referrer and user agent the audit depends on. Also check you downloaded the access log rather than the error log or a statistics file.

Analytics and Search Console set-up

The Google Analytics & Search Console vs Your Server Log section is only as good as the accounts behind it. When the comparison looks odd, check these before blaming either side.

• One Measurement ID on several sites. If two websites share a G- ID, their visitors are added together and every figure looks bigger than it is. Give each site its own property, then update the tag on the site that should move.
• No consent banner, or one that does nothing. For UK and EU visitors, analytics cookies should only load after the visitor accepts. A banner that loads the tag regardless is a compliance problem; a correct banner means analytics will count fewer people than your log, which is expected rather than a fault. See fixing cookie consent.
• Tag missing on some pages. If a template lacks the tag, analytics undercounts while the log does not. Open a few page types and check Reports → Realtime while you browse.
• No Search Console property for the site. Without one there are no search figures to compare. A Domain property verified with a DNS TXT record covers http, https, www and every subdomain in one go.
• Different date ranges or time zones. Compare the exact dates your log covers, and make sure the GA4 figures, read in the property's time zone, cover the same dates as the log.
• Links you cannot attribute. If Where People Came From is mostly Direct / none, add UTM parameters to links in emails, newsletters and app posts. They appear in the requested URL in your log as well as in analytics.

Connecting is optional, needs a free aiwebpageseo account and uses read-only access granted on Google's own sign-in page. You can disconnect at any time from the tool.

Order of work

1. Collect a decent log. Download every access log file your host keeps, including the compressed ones, so you cover at least a few weeks.
2. Run the Real Traffic Audit with the domain and every optional field filled. Server IP, your own IPs and your analytics user total for the same dates. Each one makes the sorting more accurate.
3. Clear security events first. Confirm every admin access and new account is yours; act straight away if not.
4. Fix server errors, then the most-requested 404s. Work from the top of Errors Real Visitors Hit down.
5. Clean up analytics. Add the internal traffic rule, deal with monitors and audit tools, and check for shared Measurement IDs and consent.
6. Make your AI crawler decision and write it into robots.txt deliberately.
7. Extend log retention so the next run has more to work with.
8. Re-run on a fresh log after two to four weeks with the same settings, and compare the likely people figure and the analytics gap with the first run. Download the PDF report each time so you have a record of what changed.

The Real Traffic Audit Guide explains every report section in detail if a finding is unclear.