Removing invalid/incomplete multibyte characters

Question

I'm having some issues using the following code on user input:

htmlentities($string, ENT_COMPAT, 'UTF-8');

When an invalid multibyte character is detected PHP throws a notice:

PHP Warning: htmlentities(): Invalid multibyte sequence in argument in /path/to/file.php on line 123

My first thought was to supress the error, but this is slow and poor practice: http://derickrethans.nl/five-reasons-why-the-shutop-operator-should-be-avoided.html

My second thought was to use the ENT_IGNORE flag, but even the PHP manual suggests not to use this:

Silently discard invalid code unit sequences instead of returning an empty string. Using this flag is discouraged as it » may have security implications.

A bit of further reason led me to the following piece of code:

    // detect encoding
$encoding =  mb_detect_encoding($query);
if($encoding != 'UTF-8') {
    $query = mb_convert_encoding($query, 'UTF-8', $encoding);
} else {
    // strip out invalid utf8 sequences
    $query = iconv('UTF-8', 'UTF-8//IGNORE', $query);
}

Unfortunately iconv also throws an E_NOTICE when it removes/ignores invalid characters:

If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can't be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, str is cut from the first illegal character and an E_NOTICE is generated.

So I'm basically out of options here. I'd rather use a tried and tested library to handle this kind of stuff than attempting it with a few of the regular expression based solutions I've seen floated around.

So that leads me to my final question: How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

If you don't want to use `ENT_IGNORE`, you wouldn't want to use `//IGNORE` either. They do the same thing and have the same security implications. This may be an obvious point and a lazy approach but... *shouldn't you be hiding these errors in production anyway*? The point of these situations throwing an `E_NOTICE` is so that the server administrator is aware of potential problems with the server - invalid characters would only be present if someone was sending them maliciously or some data was corrupted, both of which require administrator attention. It's an extreme edge case at any rate. — DaveRandom, Mar 09 '12 at 10:40
Is *rejecting* invalidly encoded UTF-8 an option? If it's broken, you probably shouldn't be using it to begin with. — deceze, Mar 09 '12 at 12:36
Dave, yeah the errors are hidden, but we're looking at them in the logs. It's an edge case where someone was sending bad params for one reason or another. — Dean, Mar 09 '12 at 14:03
related: http://stackoverflow.com/questions/2327219/htmlentities-invalid-multibyte-sequence-error/11664876#11664876 — Kzqai, Jul 26 '12 at 12:57
From [article](http://nikic.github.com/2012/01/28/htmlspecialchars-improvements-in-PHP-5-4.html): in PHP 5.4 thankfully this behavior is gone. The error will not be generated anymore. read this — xkeshav, Feb 28 '13 at 12:50

score 4 · Answer 1 · answered Mar 14 '13 at 16:44

4

iconv('UTF-8', "ISO-8859-1//IGNORE", $string);

worked extremely well for me. Doesn't seem to generate any notice.

answered Mar 14 '13 at 16:44

Nick Pickering

3,095
3
29
50

1

+1 I was using: 'iconv('UTF-8', 'ASCII//TRANSLIT', $var)' and the IGNORE instead of TRANSLIT fixed the 'invalid characters' notice and removed the unwanted emoji icons from the string. – Kapitein Witbaard Aug 22 '15 at 12:35

score 3 · Accepted Answer · answered Mar 10 '12 at 23:52

How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

Well, as you already have outlined in your question on your own (or at least linked), deleting the invalid byte sequence(s) is not an option.

Instead it should be probably replaced with the replacement character U+FFFD. As of PHP 5.4.0 you can make use of the ENT_SUBSTITUTE flag for htmlentities. That's probably most safe if you don't want to reject the string.

iconv will always give you warning in recent PHP versions if not even deleting the whole string. So it does not look like a good alternative for you.

Removing invalid/incomplete multibyte characters

2 Answers2

Linked