Slug function not working on my host - special characters removed

Question

I have a weird problem using a slug function to create a slug from a certain string.

function createSlug($str, $replace=array(), $delimiter='-') {
    setlocale(LC_ALL, 'en_US.UTF8');
    if( !empty($replace) ) {
        $str = str_replace((array)$replace, ' ', $str);
    }
    $clean = iconv('UTF-8', 'ASCII//TRANSLIT', $str);
    $clean = preg_replace("/[^a-zA-Z0-9\/_|+ -]/", '', $clean);
    $clean = strtolower(trim($clean, '-'));
    $clean = preg_replace("/[\/_|+ -]+/", $delimiter, $clean);
    //strip extra slashes
    $clean = str_replace('/', '-', $clean);
    return $clean;
}

echo createSlug("Abbaye de Saint-Martin Cuvée de Noël.");

result: abbaye-de-saint-martin-cuve-de-nol

However, when I upload my script to another webspace of a different host, it works fine! The output is what is should be: abbaye-de-saint-martin-cuvee-de-noel

Is there any explanation how this can happen? I just use the same script uploaded to my root folder of my host.

update:

iconv is enabled but it is not using UTF-8 and I do, but on my other hosts it works and the settings are exactly the same except for this: iconv library version 2.5

Info from phpinfo(): iconv

iconv support enabled iconv implementation glibc iconv library version 2.11.3

Directive Local Value Master Value iconv.input_encoding ISO-8859-1 ISO-8859-1 iconv.internal_encoding ISO-8859-1 ISO-8859-1 iconv.output_encoding ISO-8859-1 ISO-8859-1

it should be de-noel, the same as the original but lowercase and without accents. — randomizer, Oct 30 '13 at 18:46
It's likely a character set encoding issue. Possibly related: http://stackoverflow.com/questions/1728746/how-to-properly-handle-international-character-in-php-mysql-apache — John, Oct 30 '13 at 18:47
@John, indeed, alle the special characters are removed. The problem isn't database related. — randomizer, Oct 30 '13 at 18:49
@Sébastien, is there a way I can check this? I'm on shared hosting. — randomizer, Oct 30 '13 at 18:50
Run `phpinfo();` in a separate file. You should find a section dedicated to iconv indicating "iconv support enabled". — Sébastien, Oct 30 '13 at 18:54
@Sébastien, lol offcourse, should have known that, it is indeed enabled, I updated my question with extra info. — randomizer, Oct 30 '13 at 19:23
I tried your function on my test machine: it works. So try to output your `$clean` variable at every step to see where it breaks. — Sébastien, Oct 30 '13 at 19:35
It is indeed the iconv function which goes wrong so it seems. Think I better contact my host :) — randomizer, Oct 30 '13 at 20:53

score 0 · Answer 1 · edited May 23 '17 at 12:28

You most likely do not have iconv on the server. If you cannot easily install it you can use character translation - there are a number of solutions floating around. Here's the one that WordPress uses with your code attached. The code comes from this SO post

<?php
function seems_utf8($str)
{
    $length = strlen($str);
    for ($i=0; $i < $length; $i++) {
        $c = ord($str[$i]);
        if ($c < 0x80) $n = 0; # 0bbbbbbb
        elseif (($c & 0xE0) == 0xC0) $n=1; # 110bbbbb
        elseif (($c & 0xF0) == 0xE0) $n=2; # 1110bbbb
        elseif (($c & 0xF8) == 0xF0) $n=3; # 11110bbb
        elseif (($c & 0xFC) == 0xF8) $n=4; # 111110bb
        elseif (($c & 0xFE) == 0xFC) $n=5; # 1111110b
        else return false; # Does not match any model
        for ($j=0; $j<$n; $j++) { # n bytes matching 10bbbbbb follow ?
            if ((++$i == $length) || ((ord($str[$i]) & 0xC0) != 0x80))
                return false;
        }
    }
    return true;
}

/**
 * Converts all accent characters to ASCII characters.
 *
 * If there are no accent characters, then the string given is just returned.
 *
 * @param string $string Text that might have accent characters
 * @return string Filtered string with replaced "nice" characters.
 */
function remove_accents($string) {
    if ( !preg_match('/[\x80-\xff]/', $string) )
        return $string;

    if (seems_utf8($string)) {
        $chars = array(
        // Decompositions for Latin-1 Supplement
        chr(195).chr(128) => 'A', chr(195).chr(129) => 'A',
        chr(195).chr(130) => 'A', chr(195).chr(131) => 'A',
        chr(195).chr(132) => 'A', chr(195).chr(133) => 'A',
        chr(195).chr(135) => 'C', chr(195).chr(136) => 'E',
        chr(195).chr(137) => 'E', chr(195).chr(138) => 'E',
        chr(195).chr(139) => 'E', chr(195).chr(140) => 'I',
        chr(195).chr(141) => 'I', chr(195).chr(142) => 'I',
        chr(195).chr(143) => 'I', chr(195).chr(145) => 'N',
        chr(195).chr(146) => 'O', chr(195).chr(147) => 'O',
        chr(195).chr(148) => 'O', chr(195).chr(149) => 'O',
        chr(195).chr(150) => 'O', chr(195).chr(153) => 'U',
        chr(195).chr(154) => 'U', chr(195).chr(155) => 'U',
        chr(195).chr(156) => 'U', chr(195).chr(157) => 'Y',
        chr(195).chr(159) => 's', chr(195).chr(160) => 'a',
        chr(195).chr(161) => 'a', chr(195).chr(162) => 'a',
        chr(195).chr(163) => 'a', chr(195).chr(164) => 'a',
        chr(195).chr(165) => 'a', chr(195).chr(167) => 'c',
        chr(195).chr(168) => 'e', chr(195).chr(169) => 'e',
        chr(195).chr(170) => 'e', chr(195).chr(171) => 'e',
        chr(195).chr(172) => 'i', chr(195).chr(173) => 'i',
        chr(195).chr(174) => 'i', chr(195).chr(175) => 'i',
        chr(195).chr(177) => 'n', chr(195).chr(178) => 'o',
        chr(195).chr(179) => 'o', chr(195).chr(180) => 'o',
        chr(195).chr(181) => 'o', chr(195).chr(182) => 'o',
        chr(195).chr(182) => 'o', chr(195).chr(185) => 'u',
        chr(195).chr(186) => 'u', chr(195).chr(187) => 'u',
        chr(195).chr(188) => 'u', chr(195).chr(189) => 'y',
        chr(195).chr(191) => 'y',
        // Decompositions for Latin Extended-A
        chr(196).chr(128) => 'A', chr(196).chr(129) => 'a',
        chr(196).chr(130) => 'A', chr(196).chr(131) => 'a',
        chr(196).chr(132) => 'A', chr(196).chr(133) => 'a',
        chr(196).chr(134) => 'C', chr(196).chr(135) => 'c',
        chr(196).chr(136) => 'C', chr(196).chr(137) => 'c',
        chr(196).chr(138) => 'C', chr(196).chr(139) => 'c',
        chr(196).chr(140) => 'C', chr(196).chr(141) => 'c',
        chr(196).chr(142) => 'D', chr(196).chr(143) => 'd',
        chr(196).chr(144) => 'D', chr(196).chr(145) => 'd',
        chr(196).chr(146) => 'E', chr(196).chr(147) => 'e',
        chr(196).chr(148) => 'E', chr(196).chr(149) => 'e',
        chr(196).chr(150) => 'E', chr(196).chr(151) => 'e',
        chr(196).chr(152) => 'E', chr(196).chr(153) => 'e',
        chr(196).chr(154) => 'E', chr(196).chr(155) => 'e',
        chr(196).chr(156) => 'G', chr(196).chr(157) => 'g',
        chr(196).chr(158) => 'G', chr(196).chr(159) => 'g',
        chr(196).chr(160) => 'G', chr(196).chr(161) => 'g',
        chr(196).chr(162) => 'G', chr(196).chr(163) => 'g',
        chr(196).chr(164) => 'H', chr(196).chr(165) => 'h',
        chr(196).chr(166) => 'H', chr(196).chr(167) => 'h',
        chr(196).chr(168) => 'I', chr(196).chr(169) => 'i',
        chr(196).chr(170) => 'I', chr(196).chr(171) => 'i',
        chr(196).chr(172) => 'I', chr(196).chr(173) => 'i',
        chr(196).chr(174) => 'I', chr(196).chr(175) => 'i',
        chr(196).chr(176) => 'I', chr(196).chr(177) => 'i',
        chr(196).chr(178) => 'IJ',chr(196).chr(179) => 'ij',
        chr(196).chr(180) => 'J', chr(196).chr(181) => 'j',
        chr(196).chr(182) => 'K', chr(196).chr(183) => 'k',
        chr(196).chr(184) => 'k', chr(196).chr(185) => 'L',
        chr(196).chr(186) => 'l', chr(196).chr(187) => 'L',
        chr(196).chr(188) => 'l', chr(196).chr(189) => 'L',
        chr(196).chr(190) => 'l', chr(196).chr(191) => 'L',
        chr(197).chr(128) => 'l', chr(197).chr(129) => 'L',
        chr(197).chr(130) => 'l', chr(197).chr(131) => 'N',
        chr(197).chr(132) => 'n', chr(197).chr(133) => 'N',
        chr(197).chr(134) => 'n', chr(197).chr(135) => 'N',
        chr(197).chr(136) => 'n', chr(197).chr(137) => 'N',
        chr(197).chr(138) => 'n', chr(197).chr(139) => 'N',
        chr(197).chr(140) => 'O', chr(197).chr(141) => 'o',
        chr(197).chr(142) => 'O', chr(197).chr(143) => 'o',
        chr(197).chr(144) => 'O', chr(197).chr(145) => 'o',
        chr(197).chr(146) => 'OE',chr(197).chr(147) => 'oe',
        chr(197).chr(148) => 'R',chr(197).chr(149) => 'r',
        chr(197).chr(150) => 'R',chr(197).chr(151) => 'r',
        chr(197).chr(152) => 'R',chr(197).chr(153) => 'r',
        chr(197).chr(154) => 'S',chr(197).chr(155) => 's',
        chr(197).chr(156) => 'S',chr(197).chr(157) => 's',
        chr(197).chr(158) => 'S',chr(197).chr(159) => 's',
        chr(197).chr(160) => 'S', chr(197).chr(161) => 's',
        chr(197).chr(162) => 'T', chr(197).chr(163) => 't',
        chr(197).chr(164) => 'T', chr(197).chr(165) => 't',
        chr(197).chr(166) => 'T', chr(197).chr(167) => 't',
        chr(197).chr(168) => 'U', chr(197).chr(169) => 'u',
        chr(197).chr(170) => 'U', chr(197).chr(171) => 'u',
        chr(197).chr(172) => 'U', chr(197).chr(173) => 'u',
        chr(197).chr(174) => 'U', chr(197).chr(175) => 'u',
        chr(197).chr(176) => 'U', chr(197).chr(177) => 'u',
        chr(197).chr(178) => 'U', chr(197).chr(179) => 'u',
        chr(197).chr(180) => 'W', chr(197).chr(181) => 'w',
        chr(197).chr(182) => 'Y', chr(197).chr(183) => 'y',
        chr(197).chr(184) => 'Y', chr(197).chr(185) => 'Z',
        chr(197).chr(186) => 'z', chr(197).chr(187) => 'Z',
        chr(197).chr(188) => 'z', chr(197).chr(189) => 'Z',
        chr(197).chr(190) => 'z', chr(197).chr(191) => 's',
        // Euro Sign
        chr(226).chr(130).chr(172) => 'E',
        // GBP (Pound) Sign
        chr(194).chr(163) => '');

        $string = strtr($string, $chars);
    } else {
        // Assume ISO-8859-1 if not UTF-8
        $chars['in'] = chr(128).chr(131).chr(138).chr(142).chr(154).chr(158)
            .chr(159).chr(162).chr(165).chr(181).chr(192).chr(193).chr(194)
            .chr(195).chr(196).chr(197).chr(199).chr(200).chr(201).chr(202)
            .chr(203).chr(204).chr(205).chr(206).chr(207).chr(209).chr(210)
            .chr(211).chr(212).chr(213).chr(214).chr(216).chr(217).chr(218)
            .chr(219).chr(220).chr(221).chr(224).chr(225).chr(226).chr(227)
            .chr(228).chr(229).chr(231).chr(232).chr(233).chr(234).chr(235)
            .chr(236).chr(237).chr(238).chr(239).chr(241).chr(242).chr(243)
            .chr(244).chr(245).chr(246).chr(248).chr(249).chr(250).chr(251)
            .chr(252).chr(253).chr(255);

        $chars['out'] = "EfSZszYcYuAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy";

        $string = strtr($string, $chars['in'], $chars['out']);
        $double_chars['in'] = array(chr(140), chr(156), chr(198), chr(208), chr(222), chr(223), chr(230), chr(240), chr(254));
        $double_chars['out'] = array('OE', 'oe', 'AE', 'DH', 'TH', 'ss', 'ae', 'dh', 'th');
        $string = str_replace($double_chars['in'], $double_chars['out'], $string);
    }

    return $string;
}


function createSlug($str, $replace=array(), $delimiter='-') {
    setlocale(LC_ALL, 'en_US.UTF8');
    if( !empty($replace) ) {
        $str = str_replace((array)$replace, ' ', $str);
    }
    //$clean = iconv('UTF-8', 'ASCII//TRANSLIT', $str);
    $clean = remove_accents($str);
    $clean = preg_replace("/[^a-zA-Z0-9\/_|+ -]/", '', $clean);
    $clean = strtolower(trim($clean, '-'));
    $clean = preg_replace("/[\/_|+ -]+/", $delimiter, $clean);
    //strip extra slashes
    $clean = str_replace('/', '-', $clean);
    return $clean;
}

echo createSlug("Abbaye de Saint-Martin Cuvée de Noël.");
?>

Thx, butICONV is installed on the server, that's the weirdness. I can use some alternative functions but I would like to know what causes the problem. — randomizer, Nov 04 '13 at 10:40

score 0 · Answer 2 · answered Nov 01 '13 at 16:58

Here is the slug function I use for French UTF-8 text. It doesn't rely on iconv because I used it a long time ago on a shared hosting with no chance to install it.

Since your text is French this might help you. I have commented the function but the general idea is:

make a list of characters that I absolutely want to translit
make a corresponding list of url-safe ascii characters.
all remaining unrecognized characters are eliminated via a regex

Additionnally you can specify the word separator, you can choose to allow uppercase characters, all HTML tags are removed and all trailing and doubled word separators are removed.

function createSlug( $str = '', $allowUppercase = 0, $separator = '-' )
{
    // Illegal chars contains non-French characters that my be encountered in French text with names of foreign authors
    $illegal = array( ' ', ' ', '#160;', '&amp;', '—', '–', '\'', '/', ',', '&', '...', '?', '!', '(', ')', '&#160;', '…', '’', '.', ':', 'à', 'á', 'â', 'ã', 'ä', 'å', 'ò', 'ó', 'ô', 'õ', 'ö', 'ø', 'è', 'é', 'ê', 'ë', 'ç', 'ì', 'í', 'î', 'ï', 'ù', 'ú', 'û', 'ü', 'ÿ', 'ñ', 'œ', 'æ', '%', '"' );
    $illegalUc = array( 'À', 'Á', 'Â', 'Ã', 'Ä', 'Å', 'Ò', 'Ó', 'Ô', 'Õ', 'Ö', 'Ø', 'È', 'É', 'Ê', 'Ë', 'Ç', 'Ì', 'Í', 'Î', 'Ï', 'Ù', 'Ú', 'Û', 'Ü', 'Ÿ', 'Ñ', 'Œ', 'Æ' );

    // Corresponding list of chaaracters/words to replace illegal chars
    $legal = array( $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, $separator, 'a', 'a', 'a', 'a', 'a', 'a', 'o', 'o', 'o', 'o', 'o', 'o', 'e', 'e', 'e', 'e', 'c', 'i', 'i', 'i', 'i', 'u', 'u', 'u', 'u', 'y', 'n', 'oe', 'ae', $separator.'pourcent'.$separator, $separator );
    $legalUc = array( 'A', 'A', 'A', 'A', 'A', 'A', 'O', 'O', 'O', 'O', 'O', 'O', 'E', 'E', 'E', 'E', 'C', 'I', 'I', 'I', 'I', 'U', 'U', 'U', 'U', 'Y', 'N', 'OE', 'AE' );

    $str = strip_tags( $str ); // remove all HTML tags

    if( $allowUppercase )
    {
        $illegal = array_merge( $illegal, $illegalUc );
        $legal = array_merge( $legal, $legalUc );
    }

    if( !$allowUppercase )
    {
        $str = mb_strtolower( $str, 'UTF-8' );
    }
    $str = str_replace( $illegal, $legal, $str ); // The main translit using our 2 lists

    // Strip all other illegal characters
    $pattern = '/[^A-Za-z0-9]/ui'; // Unicode, case insensitive

    $str = preg_replace( $pattern, $separator, $str );

    // Reduce all double separators to single ones
    while( ( strpos( $str, $separator.$separator ) !== false ) )
    {
        $str = str_replace( $separator.$separator, $separator, $str );
    }

    // Remove any trailing separator
    $str = ltrim( $str, $separator );
    $str = rtrim( $str, $separator );

    return $str; // the string is returned with or without changes
}

Thx, nice function, I have done something similar now to filter the special characters but I'm still curious why my current script isn't working while ICONV is configured on the server. — randomizer, Nov 04 '13 at 10:41
Odd. And I assume that PHP is configured to use it? phpinfo(); — cyberwombat, Nov 04 '13 at 15:20

Slug function not working on my host - special characters removed

2 Answers2