Converting Window-1252 to UTF-8 Issue

Question

I have created a function to convert the following text to UTF-8, as it appeared to be in Windows-1252 format, due to being copied to a database table from a Word Document.

Testing weird character’s correction

This seems to fix the dodgy ’ character. However i'm not getting � in the following:

Devon�s most prominent dealerships

When passing the following through the same function:

Devon's most prominent dealerships

Below is the code which does the converting:

function Windows1252ToUTF8($text) {
    return mb_convert_encoding($text, "Windows-1252", "UTF-8");
}

Edit: The database can't be changed due to holding thousands of custom records. I tried the below but the mb_detect_encoding thinks character’s correction is UTF-8.

function Windows1252ToUTF8($text) {
    if (mb_detect_encoding($text) == "UTF-8") {
        return $text;
    }
    return mb_convert_encoding($text, "Windows-1252", "UTF-8");
}

Edit 2: Just tried the example from the PHP Documentation:

$str = 'áéóú'; // ISO-8859-1
echo "<pre>";
var_dump(mb_detect_encoding($str, 'UTF-8')); // 'UTF-8'
var_dump(mb_detect_encoding($str, 'UTF-8', true)); // false
echo "</pre>";
die();

but this simply outputs:

string(5) "UTF-8" string(5) "UTF-8"

So I can't even detect the encoding of the string :S

Edit 3: This seems to do the trick:

function Windows1252ToUTF8($text) {
    $badChars = [ "â", "á", "ú", "é", "ó" ];
    $match = preg_match("/[".join("",$badChars)."]/", $text);
    if ($match) {
        return mb_convert_encoding($text, "Windows-1252", "UTF-8");
    }
    return $text;
}

Edit 4: I have matched the hex values to their corresponding values. However when I get to the weird characters they don't appear to match.

Answer 1

Converting Testing weird character’s correction using bin2hex gives me 54657374696e6720776569726420636861726163746572c3a2e282ace284a27320636f7272656374696f6e

This means the "’" is actually the bytes \xc3\xa2\xe2\x82\xac\xe2\x84\xa2. This is a typical sign of a UTF-8 string having been interpreted as Windows Latin-1/1252, and then re-encoded to UTF-8.

’ (UTF-8 \xe2\x80\x99)
→ bytes interpreted as Latin-1 equal the string ’
→ characters encoded to UTF-8 result in \xc3\xa2\xe2\x82\xac\xe2\x84\xa2

To restore the original, you need to reverse that chain of mis-encodings:

$s = "\xc3\xa2\xe2\x82\xac\xe2\x84\xa2";
echo mb_convert_encoding($s, 'Windows-1252', 'UTF-8');

This interprets the string as UTF-8, converts it to the Windows-1252 equivalent, which is then the valid UTF-8 representation of ’.

Preferably you figure out at what point the encoding screwed up like this and you stop that from happening in the future. If it happened by "copy and pasting from Word", then basically somebody pasted garbage into your database and you need to fix the workflow with Word somehow. Otherwise there may be an incorrect encoding-conversion step somewhere in your code which you need to fix.

Answer 2

The following seems to do the trick. Not the way I wanted it to work by checking for specific characters, but it does the trick.

function Windows1252ToUTF8($text) {
    $badChars = [ "â", "á", "ú", "é", "ó" ];
    $match = preg_match("/[".join("",$badChars)."]/", $text);
    if ($match) {
        return mb_convert_encoding($text, "Windows-1252", "UTF-8");
    }
    return $text;
}

Edit:

function Windows1252ToUTF8($text) {
    // http://www.fileformat.info/info/charset/UTF-8/list.htm
    $illegal_hex = [ "c3a2", "c3a1", "c3ba", "c3a9", "c3b3" ];
    $match = preg_match("/".join("|",$illegal_hex)."/", bin2hex($text));
    if ($match) {
        return mb_convert_encoding($text, "Windows-1252", "UTF-8");
    }
    return $text;
}