Fixing Character Encoding Errors

Reading the symptoms

The garbled characters are not random. Each pattern points to a particular mismatch, so it is worth looking closely before changing anything.

The first five are display problems: the right bytes exist somewhere and are being misread, so fixing the declaration may be enough. The last one is data loss. Question marks saved into a database cannot be turned back into the original letters; you need a backup or the original source.

How a browser decides the encoding

A browser has to choose an encoding before it can read the page, so it checks several places in a fixed order and uses the first one it finds:

1. A byte order mark. If the file starts with the UTF-8 BOM, the browser uses UTF-8 and ignores everything else. A BOM is allowed in HTML, including before the doctype.
2. The HTTP header. A charset parameter on the Content-Type response header, for example text/html; charset=utf-8.
3. The meta tag. <meta charset="utf-8">, which the browser looks for by pre-scanning the start of the document.
4. A guess. If none of these exist, the browser guesses from the content, the user's locale and other signals. In the UK that guess is often windows-1252.

This order explains the most confusing situations. A page can have a perfect meta tag and still display mojibake, because the server's header says ISO-8859-1 and the header wins. A file can declare windows-1252 in its meta tag and display correctly as UTF-8, because an editor added a BOM. The fix is to make all three agree, not to adjust whichever one you happen to find first.

The 1024-byte rule

The browser's pre-scan for the meta tag only reads the first 1024 bytes of the document. If the charset declaration comes after a long block of inline CSS, a large comment, a set of preload links or a tracking snippet, the browser may never see it and falls back to guessing. The HTML specification requires the declaration to sit entirely within those first 1024 bytes, which is why the HTML Checker flags a late declaration even when the value is right.

The checker messages and what to do about each

The long form <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> is still valid, but it does exactly the same job as the short form. Old themes often carry both, sometimes with different values. Delete the long one.

The fix, step by step

Server and application settings

The header is the layer most often forgotten, because it is invisible in the page source. These are the usual places to set it.

# Apache (.htaccess or virtual host)
AddDefaultCharset UTF-8
AddCharset UTF-8 .html .css .js

# nginx (server block)
charset utf-8;
charset_types text/html text/css application/javascript;

# PHP (php.ini)
default_charset = "UTF-8"

# Check the result
curl -sI https://www.example.co.uk/ | grep -i content-type
content-type: text/html; charset=utf-8

Stylesheets and scripts can have encoding problems too. A CSS file containing a pound sign in a content property, or a script containing an accented string, needs to be served as UTF-8 or begin with @charset "UTF-8"; in the case of CSS. JSON responses are UTF-8 by definition.

Frameworks and platforms

• WordPress stores the site encoding under Settings, Reading on older installs and in wp-config.php as DB_CHARSET. New installs use UTF-8 and utf8mb4 by default; migrated sites from long ago may not.
• Shopify serves pages as UTF-8. Encoding problems there almost always come from content imported from a spreadsheet saved in a legacy encoding, so re-export the CSV as UTF-8 and import again.
• Node, Python and PHP frameworks generally default to UTF-8, but check any code that reads files or calls external systems, and state the encoding explicitly when opening files.
• Static site generators read source files as UTF-8. A single Markdown file saved in windows-1252 by an older editor can produce the malformed byte sequence error on one page only.

The byte order mark: allowed, but watch PHP

A UTF-8 byte order mark is three bytes at the start of a file that announce it as UTF-8. HTML allows it, and browsers treat it as the strongest signal of all. For a static HTML file it is harmless and can even help.

The problem is with server-side includes. PHP sends everything outside its tags as output, and a BOM counts. If functions.php, wp-config.php or a language file starts with a BOM, those three bytes are sent before the page, which leads to the familiar "headers already sent" warning, broken redirects and cookies that fail to set. Because the bytes are invisible, the page source just looks as if something strange is going on at the top.

# List PHP files in a theme that start with a UTF-8 BOM
grep -rlI $'^\xEF\xBB\xBF' --include=*.php wp-content/themes/larchfield-child/

Most code editors have an option to save as "UTF-8" rather than "UTF-8 with BOM". Use that for code files, and re-save any file the command above finds.

Edge cases worth knowing

• HTML entities hide the problem. Writing £ or ’ instead of the character works in any encoding, which is why some old pages look fine. It does not fix the underlying mismatch, and content added later in plain characters will still break.
• Emails and exports. Order confirmation emails, feeds and CSV downloads have their own encoding settings. A product feed with mojibake can be rejected or shown wrongly by the service reading it.
• Copied text from word processors. Pasting from a word processor brings curly quotes and non-breaking spaces. They are fine in UTF-8 but are the first characters to break when something in the chain is wrong, which makes them useful test cases.
• Search snippets. If a search engine crawled the page while it was broken, the mojibake can appear in the snippet until the page is crawled again. Request indexing in Search Console after the fix if the page is important.
• Double conversion. Running a conversion on files that are already UTF-8 damages them. Always check the current encoding first, and convert a copy.

Preventing it

• UTF-8 everywhere. Editor defaults, file storage, server headers, database tables, database connection and the meta tag should all say UTF-8, with utf8mb4 in MySQL.
• Charset first in the head. Treat the meta charset line as fixed, with nothing allowed above it inside the head.
• Test with awkward text. Keep a test page or product containing £, €, curly quotes, an em dash, an accented name and an emoji. If any of them breaks after a change, you know immediately.
• Check imports. Ask suppliers for UTF-8 CSV files, and check a sample of imported records before publishing.