Sanitise UTF-8 in PHP

Question

The function json_encode requires a valid UTF-8 string. I have a string that may be in a different encoding. I need to ignore or substitute all invalid characters to be able to convert to JSON.

It should be something very simple and robust.
The error is in a module for manual checking, so mojibake is fine.
The code responsible for fixing encoding is in a different module. (It was broken, though.) I don’t want to duplicate responsibility.

The hexadecimal representation of an example of an invalid string: 496e76616c6964206d61726b2096

My current solution:

$raw_str = hex2bin('496e76616c6964206d61726b2096');
$sane_str = @\iconv('UTF-8', 'UTF-8//IGNORE', $raw_str);

The three problems with my code:

The iconv looks little too heavy.
Many programmers don't like @.
The iconv may ignore too much: the whole string.

Any better idea?

There is similar question, Ensuring valid UTF-8 in PHP, but I don't care about conversion.

If you don't care about conversion... what are you trying to do then? — , Apr 27 '16 at 12:26
I need a valid UTF-8 string for json_encode. The valid mojibake is fine. It’s all. — Michas, Apr 27 '16 at 12:41
Honestly speaking, your solution is the cleanest I can see. If you don't want to use `@`, you might want to run the string through a encoding check, which is troublesome. — , Apr 27 '16 at 12:45

raphael75 · Answer 1 · 2016-04-27T12:53:34.697

4

You should look into mb_convert_encoding. It is able to convert text from pretty much any encoding to another. I had to use it for a similar project: http://php.net/manual/en/function.mb-convert-encoding.php

edited Apr 27 '16 at 12:53

answered Apr 27 '16 at 12:34

raphael75

2,982
4
29
44

2

More of a comment than an answer! Use comments when you are making comments _Not my downvote by the way_ – RiggsFolly Apr 27 '16 at 12:34
2

Please do not use answers to ask clarification questions. – Jay Blanchard Apr 27 '16 at 12:36
I apologize, still learning the system. Although I think that is the function he's looking for. – raphael75 Apr 27 '16 at 12:39
3

@raphael75 You should modify your answer then, in order to back it with some confidence rather than asking a question (it just leaves room for uncertainty). You'll earn + votes rather then minuses ;-) *Stack pro-tip* – Funk Forty Niner Apr 27 '16 at 12:41
1

@raphael75 Your answer makes more sense now, as per your edit ;-) – Funk Forty Niner Apr 27 '16 at 13:23

score 2 · Accepted Answer · answered Sep 11 '16 at 21:31

2

I think this is the best solution.

$raw_str = hex2bin('496e76616c6964206d61726b2096');
$sane_str = mb_convert_encoding($raw_str, 'UTF-8', 'UTF-8');

answered Sep 11 '16 at 21:31

Michas

8,534
6
38
62

score 1 · Answer 3 · edited Jul 08 '19 at 13:50

Function json_encode expects a UTF-8 encoded string. Check the encoding using a function based on the W3C recommended regex answer in Ensuring valid UTF-8 in PHP:

function encodeUtf8($string){
 if (preg_match('%^(?:
      [\x09\x0A\x0D\x20-\x7E]            # ASCII
    | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte
    | \xE0[\xA0-\xBF][\x80-\xBF]         # excluding overlongs
    | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte
    | \xED[\x80-\x9F][\x80-\xBF]         # excluding surrogates
    | \xF0[\x90-\xBF][\x80-\xBF]{2}      # planes 1-3
    | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15
    | \xF4[\x80-\x8F][\x80-\xBF]{2}      # plane 16
 )*$%xs', $string))
    return $string;
 else
    return iconv('CP1252', 'UTF-8', $string);
}

Then you could use it:

$sane_str = encodeUtf8($raw_str);

score 1 · Answer 4 · answered Apr 27 '16 at 13:11

You could use mb_detect_encoding to detect if it's not UTF-8 and then use mb_convert_encoding to convert to text to UTF-8

<?php
/**
 * Convert json blob to UTF-8
 * @param $string String to be decoded
 * @return bool|string
 */
function convert_json($string)
{
    if (ctype_print($string)) { // binary
        return false;
    }
    $from = mb_detect_encoding($string, ['auto']);
    $to = 'UTF-8';
    if ($from !== $to) {
        $string = mb_convert_encoding($string, $to, $from);
    }
    return $string;
}

Sanitise UTF-8 in PHP

4 Answers4