Issues replacing special characters in PHP string

Question

I'm trying to replace the special characters in a PHP string with normal characters (as in replace ó with o and á with a). I tried using the PHP Normalizer::normalize function as in the following code:

if (!Normalizer::isNormalized($word, Normalizer::FORM_C))
{
    echo "original: ".$word;
    $word = Normalizer::normalize($word, Normalizer::FORM_C);

    echo "\tnormalized: ".$word."<br />";
    exit; // see if it worked without having to go through every file
}

However, Normalizer::normalize returned null and the output from that code was:

original: adiós normalized:

Since this method didn't seem to be working, I went and found a function that was supposed to remove special characters. Here is the function:

function normalize ($string) {
    $table = array(
        'Š'=>'S', 'š'=>'s', 'Đ'=>'Dj', 'đ'=>'dj', 'Ž'=>'Z', 'ž'=>'z', 'Č'=>'C', 'č'=>'c', 'Ć'=>'C', 'ć'=>'c',
        'À'=>'A', 'Á'=>'A', 'Â'=>'A', 'Ã'=>'A', 'Ä'=>'A', 'Å'=>'A', 'Æ'=>'A', 'Ç'=>'C', 'È'=>'E', 'É'=>'E',
        'Ê'=>'E', 'Ë'=>'E', 'Ì'=>'I', 'Í'=>'I', 'Î'=>'I', 'Ï'=>'I', 'Ñ'=>'N', 'Ò'=>'O', 'Ó'=>'O', 'Ô'=>'O',
        'Õ'=>'O', 'Ö'=>'O', 'Ø'=>'O', 'Ù'=>'U', 'Ú'=>'U', 'Û'=>'U', 'Ü'=>'U', 'Ý'=>'Y', 'Þ'=>'B', 'ß'=>'Ss',
        'à'=>'a', 'á'=>'a', 'â'=>'a', 'ã'=>'a', 'ä'=>'a', 'å'=>'a', 'æ'=>'a', 'ç'=>'c', 'è'=>'e', 'é'=>'e',
        'ê'=>'e', 'ë'=>'e', 'ì'=>'i', 'í'=>'i', 'î'=>'i', 'ï'=>'i', 'ð'=>'o', 'ñ'=>'n', 'ò'=>'o', 'ó'=>'o',
        'ô'=>'o', 'õ'=>'o', 'ö'=>'o', 'ø'=>'o', 'ù'=>'u', 'ú'=>'u', 'û'=>'u', 'ý'=>'y', 'ý'=>'y', 'þ'=>'b',
        'ÿ'=>'y', 'Ŕ'=>'R', 'ŕ'=>'r',
    );

    return strtr($string, $table);
}

This code had no noticeable effect, however, and returned the same string that was passed in.

I'm obtaining my strings from *.txt files in Windows 7. I've never been very good at encodings, and would appreciate any help on this issue.

normalize? your converting from one letter to another (may as well swap F with K), which may render a legitimate word in to garbage. — , Jul 06 '12 at 00:11
I guess normalizing isn't the correct word, but that is my desired result -- to swap characters. — jeff, Jul 06 '12 at 00:15
It's for a word game where there are no special characters. I'm following the example of the Scrabble Dictionary and including words that have special characters but just replacing them with a-z characters (I can't just use the Scrabble Dictionary because of licensing purposes). — jeff, Jul 06 '12 at 00:29
Use iconv. Using custom tables is not accurate nor efficient. — Vincent Savard, Jul 06 '12 at 00:41

score 4 · Answer 1 · edited May 23 '17 at 12:22

There's a great tip from this page: How to remove diacritics from text? Here's my version of it:

/** Normalize a string so that it can be compared with others without being too fussy.
*   e.g. "Ádrèñålînë" would return "adrenaline"
*   Note: Some letters are converted into more than one letter, 
*   e.g. "ß" becomes "sz", or "æ" becomes "ae"
*/
function normalize_string($string) {
    // remove whitespace, leaving only a single space between words. 
    $string = preg_replace('/\s+/', ' ', $string);
    // flick diacritics off of their letters
    $string = preg_replace('~&([a-z]{1,2})(?:acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml|caron);~i', '$1', htmlentities($string, ENT_COMPAT, 'UTF-8'));  
    // lower case
    $string = strtolower($string);
    return $string;
}

It's good because, unlike the iconv method mentioned above, there's no converting between character sets (they're a minefield).

Brombomb · Accepted Answer · 2012-07-06T00:46:58.677

I copied and pasted your code into my editor and something interesting happened. Instead of getting adios I was getting adjiós. Notice the j in the middle after the d. This was coming from the 'đ'=>'dj', in the first line of the table map. Apparently, my editor changed the đ to a regular d, and then it wouldn't convert the ó. I removed this key/value pair and suddenly it worked for me. Are you sure all of your keys are correct in your editor (Does you editor accept alternative character sets?) Here is my test file (with the đ removed:

<html>
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1">
</head>
<body>
<?php

function normalize ($string) {
    $table = array(
        'Š'=>'S', 'š'=>'s', 'Ð'=>'Dj', 'Ž'=>'Z', 'ž'=>'z', 'C'=>'C', 'c'=>'c', 'C'=>'C', 'c'=>'c',
        'À'=>'A', 'Á'=>'A', 'Â'=>'A', 'Ã'=>'A', 'Ä'=>'A', 'Å'=>'A', 'Æ'=>'A', 'Ç'=>'C', 'È'=>'E', 'É'=>'E',
        'Ê'=>'E', 'Ë'=>'E', 'Ì'=>'I', 'Í'=>'I', 'Î'=>'I', 'Ï'=>'I', 'Ñ'=>'N', 'Ò'=>'O', 'Ó'=>'O', 'Ô'=>'O',
        'Õ'=>'O', 'Ö'=>'O', 'Ø'=>'O', 'Ù'=>'U', 'Ú'=>'U', 'Û'=>'U', 'Ü'=>'U', 'Ý'=>'Y', 'Þ'=>'B', 'ß'=>'Ss',
        'à'=>'a', 'á'=>'a', 'â'=>'a', 'ã'=>'a', 'ä'=>'a', 'å'=>'a', 'æ'=>'a', 'ç'=>'c', 'è'=>'e', 'é'=>'e',
        'ê'=>'e', 'ë'=>'e', 'ì'=>'i', 'í'=>'i', 'î'=>'i', 'ï'=>'i', 'ð'=>'o', 'ñ'=>'n', 'ò'=>'o', 'ó'=>'o',
        'ô'=>'o', 'õ'=>'o', 'ö'=>'o', 'ø'=>'o', 'ù'=>'u', 'ú'=>'u', 'û'=>'u', 'ý'=>'y', 'ý'=>'y', 'þ'=>'b',
        'ÿ'=>'y', 'R'=>'R', 'r'=>'r',
    );

    return strtr($string, $table);
}

$word = 'adiós';
$length = strlen($word);

echo 'original: '. $word;
echo '<br />';
echo 'normalized: '. normalize($word); 
echo '<br />';
echo 'loop: ';

for($i = 0; $i < $length; $i++) {
    echo normalize($word[$i]);
}

?>

</body>
</html>

When I loop through each character with the 'd' => 'dj' in the array map then I correctly get adjios

Apparently Sublime Text 2 doesn't support alternative character sets. Pasted the same code in Notepad++ and it worked perfectly. Thanks! — jeff, Jul 06 '12 at 01:14

Issues replacing special characters in PHP string

2 Answers2

Linked