Outputting file contents as UTF-8 leads to character encoding issues

Question

I set my header as follows:

header( 'Content-Type: text/html; charset="utf-8"' );

and then output a local file on my server to the browser using the following code-segment:

$content = file_get_contents($sPath);
$content = mb_convert_encoding($content, 'UTF-8');
echo $content;

The files I have on the server are created by lua and thus, the output of the following is FALSE (before conversion):

var_dump( mb_detect_encoding($content) );

The files contain some characters like ™ (™) etc. and these appear as plain square boxes in browsers. I've read the following threads which were suggested as similar questions and none of the variations in my code helped:

PHP File Get Contents & String Encoding (No gzip in my case, the files are plain .txts)
file_get_contents() Breaks Up UTF-8 Characters (Tried first two top-rated solutions, neither worked. The third one isn't applicable for my case)
file_get_contents() converts UTF-8 to ISO-8859-1 (No stream is there to provide as context)

There seem to be no problems when I simply use the following:

header( 'Content-Type: text/html; charset="iso-8859-1"' );
// setting path here
$content = file_get_contents($sPath);
echo $content;

The string seems to be valid iso-8859-1. Did you try a simple utf8_encode instead of mb_convert_encoding? — db-mobile, Jan 15 '14 at 08:30
@db-mobile Yes, I did try that. Result was still the square box. — hjpotter92, Jan 15 '14 at 08:36
Did you then also try to output the file without converting at all? — db-mobile, Jan 15 '14 at 08:46

score 2 · Accepted Answer · answered Jan 15 '14 at 09:52

2

There seem to be no problems when I simply use the following:
header( 'Content-Type: text/html; charset="iso-8859-1"' );
// setting path here
$content = file_get_contents($sPath);
echo $content;

So this means the file content is actually encoded in ISO-8859-1. If you want to output this as UTF-8, then explicitly convert from ISO-8859-1 to UTF-8:

$content = mb_convert_encoding($content, 'UTF-8', 'ISO-8859-1');

You always need to know what you're converting from. Just telling PHP to "convert to UTF-8" and leaving it guessing what to convert from has an undefined outcome, and in your case it does not work.

answered Jan 15 '14 at 09:52

deceze

510,633
85
743
889

But `mb_detect_encoding($content)` results in false. – hjpotter92 Jan 15 '14 at 10:00
`mb_convert_encoding($content, 'UTF-8', 'ISO-8859-1');` didn't work either. – hjpotter92 Jan 15 '14 at 10:03
Don't ever count on anything `mb_detect_encoding` says, detecting encodings is fundamentally impossible with any degree of accuracy. What you need to know first and foremost is what encoding the file is actually in. You're not going to get anywhere without knowing that. – deceze Jan 15 '14 at 10:09
`file --mime myfile.txt` gave: **text/plain; charset=unknown-8bit**. As I said, the files are being created with lua. – hjpotter92 Jan 15 '14 at 10:12
Even lua will have to write the file in *some* known, specified encoding. Figure out what that is. Maybe you can get it to directly write UTF-8. – deceze Jan 15 '14 at 10:24

Victor Bocharsky · Answer 2 · 2014-01-15T09:47:59.557

0

Check the file encoding, is it utf-8 without BOM? For example, use the notepad++ for check file encoding.

Or mayby it's usefull:

$content = file_get_contents($sPath);
$content = htmlentities($content);
echo $content;

Or try in .htaccess:

AddDefaultCharset utf-8
AddCharset utf-8 *
<IfModule mod_charset.c>
    CharsetSourceEnc utf-8
    CharsetDefault utf-8
</IfModule>

edited Jan 15 '14 at 09:47

answered Jan 15 '14 at 08:42

Victor Bocharsky

11,930
13
58
91

The command `file 02_01_2014.txt` returns: *Non-ISO extended-ASCII text, with very long lines* – hjpotter92 Jan 15 '14 at 09:18
For the conversion, you can use [iconv](http://linux.die.net/man/1/iconv), which is installed from libc package on GNU/Linux or recode. [recode](http://recode.progiciels-bpi.ca/index.html) has more options and better error handling. Maybe it is usefull for you? – Victor Bocharsky Jan 15 '14 at 09:29
I can't convert the files on disk. Sorry. :( – hjpotter92 Jan 15 '14 at 09:32
That results in nothing. No output at all. Either with `htmlentities` or `htmlspecialchars` – hjpotter92 Jan 15 '14 at 09:42
Do you have any error? If I andestand correctly, you want replace plain square boxes in browsers to a normal symbols? – Victor Bocharsky Jan 15 '14 at 09:46
Try to add `.htaccess` with content in updated answer – Victor Bocharsky Jan 15 '14 at 09:47

Outputting file contents as UTF-8 leads to character encoding issues

2 Answers2