'Malformed UTF-8 characters, possibly incorrectly encoded' in Laravel

Question

I'm using Laravel (a PHP framework) to write a service for mobile and have the data returned in JSON format. In the data result there are some fields encoded in UTF-8.

The following statement

return JsonResponse::create($data);

returns the error below

InvalidArgumentException
HELP
Malformed UTF-8 characters, possibly incorrectly encoded

Open: /var/www/html/vendor/symfony/http-foundation/Symfony/Component/HttpFoundation/JsonResponse.php
        } catch (\Exception $exception) {
            restore_error_handler();

            throw $exception;
        }

        if (JSON_ERROR_NONE !== json_last_error()) {
            throw new \InvalidArgumentException($this->transformJsonError());
        }

I've changed:

return JsonResponse::create($data);

to

return JsonResponse::create($data, 200, array('Content-Type'=>'application/json; charset=utf-8' ));

but it still isn't working.

How can I fix it?

What is `$data`? Where does it come from, what's in it, how do you know it's UTF-8 encoded? — deceze, Jun 29 '15 at 12:52
$data is array data from database. I checked $data, it have a string "兆琪" => error when return json. — Dzung Nguyen, Jun 30 '15 at 11:34
It fixed, it is my error. Have a line code not good: substr('兆琪', ...) Thanks! — Dzung Nguyen, Jul 02 '15 at 03:03

Tiago Gouvêa · Answer 1 · 2023-06-27T10:43:33.587

91

I wrote this method to handle UTF8 arrays and JSON objects as well. It works fine with array (simple and multidimensional) and will recursively convert everything from latin1 (ISO-8859-1) to UTF-8;

/**
 * Encode array from latin1 to utf8 recursively
 * @param $dat
 * @return array|string
 */
   public static function convert_from_latin1_to_utf8_recursively($dat)
   {
      if (is_string($dat)) {
         return utf8_encode($dat);
      } elseif (is_array($dat)) {
         $ret = [];
         foreach ($dat as $i => $d) $ret[ $i ] = self::convert_from_latin1_to_utf8_recursively($d);
         return $ret;
      } elseif (is_object($dat)) {
         foreach ($dat as $i => $d) $dat->$i = self::convert_from_latin1_to_utf8_recursively($d);
         return $dat;
      } else {
         return $dat;
      }
   }

Following is a sample on how to use it:

// Just pass your array or string and the UTF8 encode will be fixed
$data = convert_from_latin1_to_utf8_recursively($data);

Updated at June 2023.

utf8_encode() was deprecated in PHP 8.2, so use mb_convert_encoding() on newer versions.

So, replace utf8_encode($dat) for mb_convert_encoding($dat, 'ISO-8859-1', 'UTF-8')

Additionally, note that ini_set('default_charset', 'ISO-8859-1') may change the behavior of the mb_convert_encoding method.

edited Jun 27 '23 at 10:43

answered Jul 15 '16 at 14:27

Tiago Gouvêa

15,036
4
75
81

4

The else keyword is not needed if you have a return in the previous block. – Alonso Urbano Jun 20 '18 at 14:04
@Vladimir Nul please do not edit the code inside the original question **even if it's not having effects on the overall result of the snippet**. Thank you – Ivan Jun 20 '18 at 16:07
Worked for me on Laravel 5.6 and DB2 AS/400. Thanks! – Douglas Nassif Roma Junior Jul 30 '18 at 18:35
@Tiago Gouvea - Your solution is not working for me. Your function is converting a turkish keyword 'Ücretsiz' to 'Ãcretsiz' which is not solving any type of problem. Thanks. – Kamlesh Sep 06 '20 at 08:46
2

Note that utf8_encode() was deprecated in PHP 8.2, use mb_convert_encoding() instead. – user2867106 Jun 26 '23 at 15:38

score 84 · Answer 2 · answered Mar 29 '18 at 20:10

84

I found the answer to this problem here

Just do

mb_convert_encoding($data['name'], 'UTF-8', 'UTF-8');

answered Mar 29 '18 at 20:10

Leoncio

1,479
2
16
19

2

Thank you. Your solution work. I wrap one of the array returned with this function and regenerate json using json_encode and all works. – Apit John Ismail Mar 31 '18 at 06:57
I am trying to convert a Turkish keyword into UTF-8 but it does not work with ajax request in laravel framework according to your solution. Thanks dear. – Kamlesh Sep 06 '20 at 08:54
This worked for me. But can anyone explain why? I mean converting from utf8 to utf8 seems like doing nothing from the outside – Steve Kirsch Jun 13 '23 at 19:33

score 9 · Answer 3 · answered Apr 13 '18 at 10:07

9

In my case I had a ucfirst on the asian letters string. This was not possible and produced a non utf8 string.

answered Apr 13 '18 at 10:07

alex

1,999
3
16
18

been searching for ages this solved my problem – Zein Mar 23 '22 at 09:28

score 7 · Answer 4 · answered Apr 08 '21 at 08:41

7

In Laravel 7.x, this helped me to get rid of this error.

$newString = mb_convert_encoding($arr, "UTF-8", "auto");
return response()->json($newString);

answered Apr 08 '21 at 08:41

Murad

1,064
16
13

This doesn't work because it turns characters like "ó" to "?" – Gabriel I. Nov 23 '22 at 22:31

score 4 · Answer 5 · answered Nov 07 '17 at 11:02

In my case, this causes error:

return response->json(["message" => "Model status successfully updated!", "data" => $model], 200);

but this not:

return response->json(["message" => "Model status successfully updated!", "data" => $model->toJson()], 200);

ThiagoYou · Answer 6 · 2019-11-18T13:01:09.313

I know it's already an old question, but i had the same error today. For me setting the connection variable on model did the work.

/**
 * Table properties
 */
protected $connection = 'mysql-utf8';
protected $table = 'notification';
protected $primaryKey = 'id';

I don't know if the issue was with the database (probably), but the texts fields with special chars (like ~, ´ e etc) were all messed up.

---- Editing

That $connection var is used to select wich db connection your model will use. Sometimes it happens that in database.php (under /config folder) you have multiples connections and the default one is not using UTF-8 charset.

In any case, be sure to properly use charset and collation into your connection.

'connections' => [

    'mysql' => [
        'driver' => 'mysql',
        'host' => env('DB_HOST', '127.0.0.1'),
        'port' => env('DB_PORT', '3306'),
        'database' => env('DB_DATABASE', 'your_database'),
        'username' => env('DB_USERNAME', 'root'),
        'password' => env('DB_PASSWORD', 'database_password'),
        'unix_socket' => env('DB_SOCKET', ''),
        'prefix' => '',
        'strict' => false,
        'engine' => null
    ],

    'mysql-utf8' => [
        'driver' => 'mysql',
        'host' => env('DB_HOST', '127.0.0.1'),
        'port' => env('DB_PORT', '3306'),
        'database' => env('DB_DATABASE', 'your_database'),
        'username' => env('DB_USERNAME', 'root'),
        'password' => env('DB_PASSWORD', 'database_password'),
        'unix_socket' => env('DB_SOCKET', ''),
        'charset' => 'utf8',
        'collation' => 'utf8_unicode_ci',
        'prefix' => '',
        'strict' => false,
        'engine' => null
    ],

Here you are setting a connection (a database) where I guess you are trying to set a special config to the database. This won't work (unless you have a db connection with that name). — Kenny Horna, Nov 16 '19 at 17:40

score 3 · Answer 7 · answered Mar 21 '19 at 00:23

3

I've experienced the same problem. The thing is that I forgot to start the apache and mysql in xampp... :S

answered Mar 21 '19 at 00:23

Wal Heredia

401
1
6
14

score 3 · Answer 8 · edited Jun 01 '21 at 01:09

For more solution i have completed the solution of (the great) Tiago Gouvêa exposed before only for strings and arrays

i used md_convert_encoding() function instead of utf8_encode(), it works for me : (12 hours loosed ...)

// this object return me a big array who have multiple arrays imbricated

$get_days = program::get_days($ARR, $client);

// and i use this function for well parsing what the server return *

function convert_to_utf8_recursively($dat){
    if( is_string($dat) ){
        return mb_convert_encoding($dat, 'UTF-8', 'UTF-8');
    }
    elseif( is_array($dat) ){
        $ret = [];
        foreach($dat as $i => $d){
            $ret[$i] = convert_to_utf8_recursively($d);
        }
        return $ret;
    }
    else{
        return $dat;
    }
}
                
// use 
$data = convert_to_utf8_recursively($get_days);

what the server return * i talk to that because i test the same code in two different servers the first return me a well formatted json , without any function, as we usually do BUT the second server does not send me back anything if I do not apply this function ...

score 1 · Answer 9 · answered Aug 31 '21 at 13:38

1

Found an answer here.

The solution turned out to be in my application code, it has nothing to do with Laravel itself.

return mb_strtoupper(mb_substr($this->first_name, 0, 1) . mb_substr($this->last_name, 0, 1));

https://barryvanveen.nl/blog/67-solving-malformed-utf-8-characters-possibly-incorrectly-encoded-error-in-laravel

answered Aug 31 '21 at 13:38

Lara

96
1
10

I had a similar issue. In short, using a non multi-byte function, such as `substr` on a UTF-8 string was incorrect. I had to use the multi-byte version instead, `mb_substr`. – caoquendo Jun 22 '23 at 13:34

starter-dev · Answer 10 · 2022-02-26T19:05:44.500

In my case, it happened twice:

I forgot to migrate again. I roll back (php artisan rollback) the latest migration operation because I want to change something on my database schema but I forgot to execute again the migrate command: php artisan migrate.

More info on migrations: https://laravel.com/docs/9.x/migrations

It also happened to me when the API is returning JSON responses with attributes of type blob (column on the database that is created with the binary() method). For my use case, I hide that attribute by adding the attribute's name to the model's array property $hidden. It works fine after that.

More info on hiding attributes on JSON: https://laravel.com/docs/9.x/eloquent-serialization#hiding-attributes-from-json

score 1 · Answer 11 · answered May 31 '23 at 08:31

1

In my case one of the MySql columns was of type POINT which the json parser could not parse. After excluding the column or adding the relevant model accessor, solved it for me.

answered May 31 '23 at 08:31

Hmerman6006

1,622
1
20
45

score 0 · Answer 12 · answered Nov 12 '18 at 08:24

0

I got this error and i fixed the issue with iconv function like following:

iconv('latin5', 'utf-8', $data['index']);

answered Nov 12 '18 at 08:24

Bora

10,529
5
43
73

score 0 · Answer 13 · answered Apr 08 '22 at 13:20

This was my problem and this is how I solved it, hope it works for someone else.

Make sure the encoding of the file is UTF-8, in vscode is in the right bottom part.

I had this particular problem when I modified a file that already was in a development server, somehow the encoding when I edited that file in that server was changed to ANSI, so, make sure too that your file is encoded in UTF-8 in the server where you are doing your deployments.

score -1 · Answer 14 · answered Dec 22 '19 at 18:20

-1

Set the charset at after you made the connection to db like

// Create connection
$conn = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

if (!$conn->set_charset("utf8")) {
    printf("Error loading character set utf8: %s\n", $conn->error);
    exit();
} else {
    printf("Current character set: %s\n", $conn->character_set_name());
}

answered Dec 22 '19 at 18:20

Tikam Chand

94
1
10

Thanks for sharing this solution. This fixed it for me. Apparently the MySQL database in our project is returning the data with a different charset. Of course I removed the ELSE line at the end of your suggested code. – ideaztech Sep 24 '21 at 12:01

score -1 · Answer 15 · answered Aug 27 '20 at 14:04

-1

First thing Correctly specify all settings in the .ENV file for Mail.

I LOST 6 HOURS FOR THIS

Example

MAIL_DRIVER=smtp
MAIL_MAILER=smtp
MAIL_HOST=smtp.yandex.ru
MAIL_PORT=465
MAIL_USERNAME=yourmail@yandex.ru
MAIL_PASSWORD=password //Create password for Apps in settings. NOT PASTE YOUR REAL MAIL PASSWORD
MAIL_ENCRYPTION=SSL
MAIL_FROM_ADDRESS=yourmail@yandex.ru
MAIL_FROM_NAME="${APP_NAME}"

answered Aug 27 '20 at 14:04

Biblbroks42

330
3
11

I changed in some values with spaces simple `'some value'` for `"some value"` and the error was gone – Pathros Sep 13 '22 at 17:08

score -1 · Answer 16 · answered Jul 19 '21 at 17:15

I experienced the same issue. However, in my case I had the column in the table specified as varchar(255) and my content was larger when I was doing inserts. So it used to truncate the part which could not fit in the varchar column size.

To fix this, I updated my column type to longtext so larger text can be stored. Also, added a frontend validation to not have text larger than the column. (I was using this to store draft js escaped HTML)

score -1 · Answer 17 · answered Apr 11 '22 at 13:52

In my case, the error was similar to what @dev1234 mentioned here (https://stackoverflow.com/a/71279140/7880201) as part of the suggested answers.

But the problem was as a result of laravel outputing the entire error which contained blob(s). I would advise you do not return just the error message in the case of

try {
} catch (\Throwable $th) {

    // $code = $th->getCode();
    $msg = $th->getMessage();
    // dd($msg);
    abort(404, 'THERE WAS AN ERROR: ' . $msg);

}

So to rectify this issue, just show a little friendly error even if you want to show it to yourself alone as a dev.

Check all blobs to ensure that laravel is not just outputing them from the ajax response and you will be fine.

I hope it saves you today. If not, there are still good answers to try. :)

'Malformed UTF-8 characters, possibly incorrectly encoded' in Laravel

17 Answers17

Linked