Fixing Canonical and URL Conflicts

One page, many addresses

To a person, example.co.uk/boilers and https://www.example.co.uk/boilers/ are obviously the same page. To software they are different URLs that happen to return similar content. Most sites quietly accept many variations of each address, and every one of them is a possible copy:

• http and https versions
• with and without www
• with and without a trailing slash
• upper and lower case letters in the path, on servers that treat them the same
• /index.html or /index.php on the end
• tracking parameters such as ?utm_source=newsletter, and sorting or filter parameters
• the same product reached through different category paths
• old staging or preview domains that still serve the site

Having variants is normal. The conflict arises when the site does not answer clearly which variant is the real one, or answers differently in different places. The Agent Readiness Audit flags pages where those declarations disagree, because a crawler that meets the disagreement has to guess.

Every place your site names a URL

There is no single canonical setting. The preferred address is stated, directly or indirectly, in several places, and they are often maintained by different people or plugins. All of them need to name the same URL.

Search engines combine these hints to choose one address, and Google may pick a different one from the one you declared when the hints disagree. Nobody outside the AI companies knows exactly how each AI crawler handles canonical tags, and it is safest to assume some treat each address they fetch as its own page. That is why redirects matter most: they are the only signal every client obeys, because the client never receives the duplicate at all.

Why this matters more for AI crawlers

Google has spent many years building systems to cluster duplicate pages. AI crawlers are newer, and their operators publish much less about how they handle duplicates. Three practical effects follow.

1. Wasted visits. A crawler that fetches the same article at five addresses has spent five requests on one page, and may not reach pages it would otherwise have read.
2. Addresses you did not choose. When an answer engine links to your page, it links to the address it knows. If that is a tracking URL from a newsletter or an old http address, readers arrive through a redirect, or not at all if the variant no longer works.
3. Stale copies. If an old variant still serves an outdated version, for example a cached page on a staging host, the crawler can quote content you have since corrected.

Finding the conflicts

Follow the redirects

Request each common variant and watch where it ends up. The -L option follows redirects and -I shows only headers, so each hop appears in turn:

curl -sIL http://example.co.uk/boilers | grep -iE "^HTTP|^location"
curl -sIL https://example.co.uk/boilers/ | grep -iE "^HTTP|^location"
curl -sIL https://www.example.co.uk/Boilers | grep -iE "^HTTP|^location"

Every variant should reach the preferred address in one hop and finish on a 200. Two or three hops in a row, such as http to https, then non-www to www, then adding the slash, form a chain. A variant that returns 200 without redirecting is an unhandled duplicate.

Read the declared canonical

curl -s https://www.example.co.uk/boilers/ | grep -io '<link[^>]*canonical[^>]*>'
curl -sI https://www.example.co.uk/files/price-list.pdf | grep -i "^link"

Check that exactly one canonical tag appears, that it is an absolute https address, and that it matches the URL you just requested. Then request a parameter version, such as ?utm_source=test, and check the canonical still names the clean address.

Compare with the sitemap and structured data

Search the sitemap for the page and compare the <loc> character by character with the canonical. Do the same for og:url and the url in any JSON-LD. Trailing slashes and www are the usual differences.

Ask Google which address it chose

In Search Console, the URL Inspection tool shows the user-declared canonical and the Google-selected canonical. When they differ, Google has decided your other signals outweigh the tag, which is a clear sign they disagree. The Pages report also lists pages marked as duplicates where Google chose a different canonical.

Common conflicts and what fixes them

Redirect rules that settle host and protocol

Host and protocol variants are best handled once, at the server or CDN, so each wrong combination reaches the right address in a single hop. On nginx, a separate server block for the non-preferred host does this cleanly:

server {
    listen 80;
    listen 443 ssl;
    server_name example.co.uk www.example.co.uk;
    # certificate directives for both names go here
    if ($host = example.co.uk) { return 301 https://www.example.co.uk$request_uri; }
    if ($scheme = http)        { return 301 https://www.example.co.uk$request_uri; }
    # normal site configuration continues below
}

On Apache with .htaccess, one rule catches both http and the bare domain, sending them straight to the final address:

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^ https://www.example.co.uk%{REQUEST_URI} [L,R=301]

If your hosting has a CDN in front, it may be simpler to set these redirects in the CDN's rules, which act before the request reaches your server. Either way, avoid doing it in both places, which is how chains start. Trailing slash and case handling usually belong in the CMS or framework, because they depend on how routes are defined.

For tracking and sorting parameters, you normally do not redirect: the page must still work with them. Instead, keep the canonical tag pointing at the clean address and avoid using parameter URLs in your own internal links and sitemap. Google retired the Search Console URL Parameters tool in 2022, so there is no setting there to lean on.

The fix, step by step

Stopping conflicts coming back

• One source of truth in templates. Build the canonical tag, og:url, JSON-LD url and hreflang addresses from the same variable in code, never typed separately.
• Test redirects after infrastructure changes. Moving host, adding a CDN or changing SSL settings is when chains and loops appear. Keep a short list of variants to curl after each change.
• Check new plugins and apps. Anything that outputs head tags may add a second canonical. View the source after installing.
• Keep links in content clean. Editors often paste addresses copied from the browser, complete with tracking parameters. A periodic search for utm_ in your content catches them.
• Plan migrations properly. Moving URLs is its own project with its own risks; the canonical migration guide covers redirect maps and checks.