file_get_contents() Breaks Up UTF-8 Characters

Question

I am loading a HTML from an external server. The HTML markup has UTF-8 encoding and contains characters such as ľ,š,č,ť,ž etc. When I load the HTML with file_get_contents() like this:

$html = file_get_contents('http://example.com/foreign.html');

It messes up the UTF-8 characters and loads Å, ¾, ¤ and similar nonsense instead of proper UTF-8 characters.

How can I solve this?

UPDATE:

I tried both saving the HTML to a file and outputting it with UTF-8 encoding. Both doesn't work so it means file_get_contents() is already returning broken HTML.

UPDATE2:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="sk" lang="sk">
<head>

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="Content-Style-Type" content="text/css" />
<meta http-equiv="Content-Language" content="sk" />
<title>Test</title>

</head>
<body>


<?php

$html = file_get_contents('http://example.com');
echo htmlentities($html);

?>

</body>
</html>

I'm not outputting it. I save it to a file and then read it. But it's irrelevant because I tried outputting it with UTF-8 and it's still messed up. — Richard Knop, Feb 10 '10 at 12:24
Re your 2nd example, you need to pass the charset to htmlentities: http://de3.php.net/htmlentities — Pekka, Feb 10 '10 at 12:49
And a guess, could it be that the remote server say `utf-8` in the meta tags but sends `iso-8859-1` in the content-type header? — Pekka, Feb 10 '10 at 12:50
Hi Pekka, actually I found out file_get_contents() is not causing this. This is caused by DOM. I do some parsing of the HTML with DOM before outputting it and I didn't think that could cause the issue. — Richard Knop, Feb 10 '10 at 13:06

score 118 · Answer 1 · edited Dec 15 '20 at 17:34

118

I had similar problem with polish language

I tried:

$fileEndEnd = mb_convert_encoding($fileEndEnd, 'UTF-8', mb_detect_encoding($fileEndEnd, 'UTF-8', true));

I tried:

$fileEndEnd = utf8_encode ( $fileEndEnd );

I tried:

$fileEndEnd = iconv( "UTF-8", "UTF-8", $fileEndEnd );

And then -

$fileEndEnd = mb_convert_encoding($fileEndEnd, 'HTML-ENTITIES', "UTF-8");

This last worked perfectly !!!!!!

edited Dec 15 '20 at 17:34

tirenweb

30,963
73
183
303

answered Mar 03 '13 at 08:20

ugniesdebesys

1,197
1
7
2

it gives me ã insetead of ă and ª instead of Ș :(( – artur99 Sep 27 '15 at 18:29
For all germans use iso-8859-1 instead of UTF-8. This will fix äöüß for you. Great fix thanks. – JustRandom Feb 22 '16 at 19:51
THIS should be the answer! For me, file_get_contents() was converting £ to the unicode version. Using mb_convert_encoding() after using file_get_contents() resolved the issue. Thank you! – Reado May 09 '17 at 09:04
Perfect : `$fileEndEnd = mb_convert_encoding($fileEndEnd, 'HTML-ENTITIES', "UTF-8");` – Peter Artoung Jan 09 '19 at 15:04
This answer led me to a solution (to correctly display ***French-Canadian characters from a `.csv` from StatsCanada***) which was to use [`mb_convert_encoding`](https://www.php.net/manual/function.mb-convert-encoding.php) like: `$myString=mb_convert_encoding($myString, 'UTF-8', "ISO-8859-1");` . . . with hints from [this](https://stackoverflow.com/a/275448/8112776) answer and [this](https://www.php.net/manual/mbstring.supported-encodings.php) list of Supported Character Encodings. Thanks! – ashleedawg Jul 20 '19 at 08:09

score 91 · Answer 2 · edited Mar 22 '17 at 13:11

91

Solution suggested in the comments of the PHP manual entry for file_get_contents

function file_get_contents_utf8($fn) {
     $content = file_get_contents($fn);
      return mb_convert_encoding($content, 'UTF-8',
          mb_detect_encoding($content, 'UTF-8, ISO-8859-1', true));
}

You might also try your luck with http://php.net/manual/en/function.mb-internal-encoding.php

edited Mar 22 '17 at 13:11

Adam

5,091
5
32
49

answered Feb 10 '10 at 12:26

Gordon

312,688
75
539
559

4

This should be marked as best answer. Thanks Gordon. – helpse Jul 21 '14 at 16:54
Easy, simple, perfect. – dynamichael Sep 08 '18 at 20:57

score 7 · Accepted Answer · edited May 23 '17 at 11:55

7

Alright. I have found out the file_get_contents() is not causing this problem. There's a different reason which I talk about in another question. Silly me.

See this question: Why Does DOM Change Encoding?

edited May 23 '17 at 11:55

Community

1
1

answered Feb 10 '10 at 13:05

Richard Knop

81,041
149
392
552

file_get_contents() is causing the problem. I had a JSON file I was opening with file_get_contents() but upon doing a print_r() after loading the JSON, the unicode characters were there but not in the JSON. Performing the mb_convert_encoding() on the file_get_contents() fixed the problem. – Reado May 09 '17 at 09:02
1

`$string = mb_convert_encoding($string, 'HTML-ENTITIES', "UTF-8");` solved it for me. – WEBjuju Feb 24 '18 at 22:16

score 5 · Answer 4 · answered Sep 17 '19 at 09:26

5

Exemple :

$string = file_get_contents(".../File.txt");
$string = mb_convert_encoding($string, 'UTF-8', "ISO-8859-1");
echo $string;

answered Sep 17 '19 at 09:26

Dorian PIERREFEU

51
1
1

How is this different from [this answer](https://stackoverflow.com/a/2236698/3558960)? – Robby Cornelissen Sep 17 '19 at 09:31

score 4 · Answer 5 · answered Nov 10 '12 at 18:59

I think you simply have a double conversion of the character type there :D

It may be, because you opened an html document within a html document. So you have something that looks like this in the end

<!DOCTYPE html> 
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title></title>
</head>
<body>
<!DOCTYPE html> 
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Test</title>.......

The use of mb_detect_encoding therefore may lead you to other issues.

score 3 · Answer 6 · answered Oct 26 '16 at 08:24

İn Turkish language, mb_convert_encoding or any other charset conversion did not work.

And also urlencode did not work because of space char converted to + char. It must be %20 for percent encoding.

This one worked!

   $url = rawurlencode($url);
   $url = str_replace("%3A", ":", $url);
   $url = str_replace("%2F", "/", $url);

   $data = file_get_contents($url);

score 3 · Answer 7 · answered Aug 31 '20 at 13:59

3

I managed to solve using this function below:

function file_get_contents_utf8($url) {
    $content = file_get_contents($url);
    return mb_convert_encoding($content, "HTML-ENTITIES", "UTF-8");
}

file_get_contents_utf8($url);

answered Aug 31 '20 at 13:59

Kário Freire

31
2

score 2 · Answer 8 · answered Nov 19 '14 at 13:55

2

Try this too

 $url = 'http://www.domain.com/';
    $html = file_get_contents($url);

    //Change encoding to UTF-8 from ISO-8859-1
    $html = iconv('UTF-8', 'ISO-8859-1//TRANSLIT', $html);

answered Nov 19 '14 at 13:55

Mohammad H.

856
9
19

score 0 · Answer 9 · answered Nov 15 '17 at 10:49

0

I am working with 35000 lines of data.

$f=fopen("veri1.txt","r");
$i=0;
while(!feof($f)){
    $i++;
    $line=mb_convert_encoding(fgets($f), 'HTML-ENTITIES', "UTF-8");
    echo $line;
}

This code convert my strange characters into normal.

answered Nov 15 '17 at 10:49

matasoy

659
1
6
7

score 0 · Answer 10 · answered Oct 18 '19 at 01:42

I had a similar problem, what solved it was html_entity_decode.

My code is:

$content = file_get_contents("http://example.com/fr");
$x = new SimpleXMLElement($content);
foreach($x->channel->item as $entry) {
    $subEntry = html_entity_decode($entry->description);
}

In here I am retrieving an xml file (in French), that's why I'm using this $x object variable. And only then I decode it into this variable $subEntry.

I tried mb_convert_encoding but this didn't work for me.

score 0 · Answer 11 · answered Jan 22 '20 at 19:55

Try this function

function mb_html_entity_decode($string) {
if (extension_loaded('mbstring') === true)
{
    mb_language('Neutral');
    mb_internal_encoding('UTF-8');
    mb_detect_order(array('UTF-8', 'ISO-8859-15', 'ISO-8859-1', 'ASCII'));

    return mb_convert_encoding($string, 'UTF-8', 'HTML-ENTITIES');
}

return html_entity_decode($string, ENT_COMPAT, 'UTF-8');

}

file_get_contents() Breaks Up UTF-8 Characters

11 Answers11

Linked

Related