Stuck writing UTF-8 file via PHP's fwrite

Question

I can't figure out what I'm doing wrong. I'm getting file content from the database. When I echo the content, everything displays just fine, when I write it to a file (.html) it breaks. I've tried iconv and a few other solutions, but I just don't understand what I should put for the first parameter, I've tried blanks, and that didn't work very well either. I assume it's coming out of the DB as UTF-8 if it's echoing properly. Been stuck a little while now without much luck.

function file($fileName, $content) {
    if (!file_exists("out/".$fileName)) {
        $file_handle = fopen(DOCROOT . "out/".$fileName, "wb") or die("can't open file");
        fwrite($file_handle, iconv('UTF-8', 'UTF-8', $content));
        fclose($file_handle);
        return TRUE;
    } else {
        return FALSE;
    }
}

Source of the html file looks like.

Comes out of the DB like this:

<h5>Текущая стабильная версия CMS</h5>

goes in file like this

<h5>Ð¢ÐµÐºÑƒÑ‰Ð°Ñ ÑÑ‚Ð°Ð±Ð¸Ð»ÑŒÐ½Ð°Ñ Ð²ÐµÑ€ÑÐ¸Ñ CMS</h5>

EDIT:

Turns out the root of the problem was Apache serving the files incorrectly. Adding

AddDefaultCharset utf-8

To my .htaccess file fixed it. Hours wasted... At least I learned something though.

Put your output HTML somewhere. Chances are you have not added the UTF-8 tag header to the HTML. — Dean, Nov 19 '15 at 22:01
@Serhiy can you add the code in your comment at the bottom of your original post? comment codes/scramble is hard enough to read at the best of times, cheers — Martin, Nov 19 '15 at 22:36
can you post a couple of comparison codeblocks of how the test looks on your output from database to how the text looks on your dump into the file? — Martin, Nov 19 '15 at 22:45
I bet that Ð¢ÐµÐºÑƒÑ‰Ð°Ñ ÑÑ‚Ð°Ð±Ð¸Ð»ÑŒÐ½Ð°Ñ Ð²ÐµÑ€ÑÐ¸Ñ is the UTF-8 of Текущая стабильная версия seen as ISO-8859-1, also because the character Ð is 0xD0 in ISO-8859-1, and 0xD0 is the first of the 2 UTF-8 bytes of the letters of the cyrillic alphabet. This means that you ARE writing UTF-8, but are looking at it as if it were ISO-8859-1 (or ISO-8859-15). — Walter Tross, Nov 19 '15 at 23:38
ahhhh, I see your edit, ok - so then it's the character encoding of the source code reader that needs to be corrected... — Martin, Nov 19 '15 at 23:40
@Walter Tross my god, I think we're getting close, when I dive into Putty and cat the file, it looks ok, so is an Apache issue? and where would I even begin with that... — Serhiy, Nov 19 '15 at 23:48
@Martin this just gets better and better, in chrome everything appears fine, in IE and firefox it's scrambled. — Serhiy, Nov 19 '15 at 23:50
Man ohh man was this ever frustrating. @WalterTross thank you for pointing me in the right direction. Martin thank you so much for all the debugging help and helping me eliminate all the possible problems. I feel like I should be paying you, you offered me so much help. — Serhiy, Nov 19 '15 at 23:55
did you get a solution? - I have no immediate ideas about firefox (etc) scrambling the encoding.... maybe revert back to what Dean said and add a encoding type header. — Martin, Nov 19 '15 at 23:55
@Martin if you want to integrate it into your answer, I will accept it, since you've been so helpful. — Serhiy, Nov 19 '15 at 23:56
ahhhh `.htaccess` . Really pleased to get a solution to this, it was confusing me. I don't want to take the credit, you discovered this yourself. Just mark up (+1) my answer ;-) — Martin, Nov 19 '15 at 23:58
Does the `.htaccess` issue also correct the FF / IE browsers, too? — Martin, Nov 19 '15 at 23:59
awesome! As I say, it's satisfying as a helper trying to work out what's going on to discover (be told) the solution. And just in time, now I need to disappear! Finally, though, what made you think of adding that line to .htaccess? — Martin, Nov 20 '15 at 00:04
@Martin The Walter Tross comment you pointed me towards. He said it's the reader. So after I outputed the file via cat, I got good results, and figured my reader, aka Apache, was at fault. Next thing I did was google Apache and UTF-8 and got http://stackoverflow.com/questions/913869/how-to-change-the-default-encoding-to-utf-8-for-server — Serhiy, Nov 20 '15 at 00:16
@Martin Should this be of use, finished the thing I was doing https://github.com/3rdcupofjava/scraping_old_kohana_forum all the code is actually in this file https://github.com/3rdcupofjava/scraping_old_kohana_forum/blob/master/application/classes/Controller/Main.php — Serhiy, Nov 20 '15 at 02:19
Glad I could help. I have to correct my comment though: Characters in the cyrillic block of Unicode start with one of 0xD0, 0xD1, 0xD2, 0xD3 when encoded as UTF-8 (because they are in the range U+0400 to U+04FF). These 4 start bytes appear as Ð, Ñ, Ò and Ó in ISO-8859-1 (with decreasing frequency). — Walter Tross, Nov 20 '15 at 07:16

score 1 · Answer 1 · edited May 23 '17 at 12:15

1

Edit: The database encoding does not seem to be the issue here, so this part of the answer is retained for information only

I assume it's coming out of the DB as UTF-8

This is most likely your problem, what database type do you use? Have you set the character encoding and collation details for the database, the table, the connection and the transfer.

If I was to hazard a guess, I would say your table is MySQL and that your MySQL collation for the database / table / column should all be UTF8_general_ci ?

However, for some reason MySQL UTF8 is not actually UTF8, as it stores its data in 3bits rather than 4bits, so can not store the whole UTF-8 Character sets, see UTF-8 all the way through .

So you need to go through every table, column on your MySQL and change it from UTF8_ to the UTF8mb4_ (note: since MySQL 5.5.3) which is UTF8_multibyte_4 which covers the whole UTF-8 Spectrum of characters.

Also if you do any PHP work on the data strings be aware you should be using mb_ PHP functions for multibyte encodings.

And finally, you need to specify a connection character set for the database, don't run with the default one as it will almost certainly not be UTF8mb4, and hence you can have the correct data in the database, but then that data is repackaged as 3bit UTF8 before then being treated as 4bit UTF8 by PHP at the other end.

Hope this helps, and if your DB is not MySQL, let us know what it is!

Edit:

function file($fileName, $content) {
    if (!file_exists("out/".$fileName)) {
        $file_handle = fopen(DOCROOT . "out/".$fileName, "wb") or die("can't open file");
        fwrite($file_handle, iconv('UTF-8', 'UTF-8', $content));
        fclose($file_handle);
        return TRUE;
    } else {
        return FALSE;
    }
}

your $file_handle is trying to open a file inside an if statement that will only run if the file does not exist.
Your iconv is worthless here, turning from "utf-8" to er, "utf-8". character detection is extremely haphazard and hard for programs to do correctly so it's generally advised not to try and work out / guess what a character encoding it, you need to know what it is and tell the function what it is.

edited May 23 '17 at 12:15

Community

1
1

answered Nov 19 '15 at 22:32

Martin

22,212
11
70
132

Arg, still failing, switched the table / connection to be UTF8mb4, re-inserted the data just to make sure, commented out the str_replace just in case, took out the iconv, made "wb" just "w" and am still failing. – Serhiy Nov 19 '15 at 22:59
you will need to check that the connection between PHP and MySQL is also utf8mb4, something like `$dbconnection->set_charset("utf8mb4");` – Martin Nov 19 '15 at 23:02
if that also doesn't help you, then we'd need to see what the code looks like on your screen output (when it's ok) and compare that to what it looks like on your file (which you put in your question already). Also if you have changed the database to use UTF8mb4 then you will ideally need to re-add the code to the database so it can be saved in 4bit rather than 3bit UTF8. @Serhiy – Martin Nov 19 '15 at 23:04
Yep, re-inserted the data, had the set_charset thing, when I echo, everything is perfect, when I write to file, it's not. I honestly feel like it must be the file writing part. I am initially getting the data from CURL, but since it's getting in the DB and out just fine, there's no way that could be the culprit could it? http://pastebin.com/raw.php?i=EshBmV77 – Serhiy Nov 19 '15 at 23:07
what is the character encoding if your output HTML file when it works? I take it that is UTF-8 (I'm just ticking off causes it isn't :-D ) – Martin Nov 19 '15 at 23:08
do you do anything to `$content` between filling the variable with the MySQL data source and sending it to the fopen? I would suggest replacing the `b` if it doesn't fix it removing it . I take it you are using a windows based system, If not then keep the `b` gone :) @Serhiy – Martin Nov 19 '15 at 23:10
Removed the b a while ago, I code in windows, but my dev box is ubuntu server 14 – Serhiy Nov 19 '15 at 23:13
unfortunately I know very little about ubuntu :( – Martin Nov 19 '15 at 23:19
I'm a bit confused about your last statement, I'm not reading from files, just writing them. Static .html pages. I publish my code to github, it's not anything private. It's just a bit much because I tossed it into a Kohana framework. Though I don't see how that could be a problem. – Serhiy Nov 19 '15 at 23:20
sorry my mistake. I deleted that comment now. Ok -- the source of the error can only be then if something happening with the data in the PHP page. You can use the http://php.net/manual/en/function.mb-detect-encoding.php function to see if you can get the character encoding of the string before it's written to the file? If that's ok then it's the filesystem causing an issue and not saving the file correctly... (which was your original statement!) @Serhiy – Martin Nov 19 '15 at 23:23
on ubuntu you don't need the `b` :-) – Martin Nov 19 '15 at 23:23
still tried it, switched to file_puts_content same error. Using mb_detect encoding spits out ASCII ASCII UTF-8 UTF-8 UTF-8 UTF-8 which corresponds correctly to which charset which page uses. http://pastebin.com/raw.php?i=aqbguhDZ – Serhiy Nov 19 '15 at 23:31
@Serhiy Without any further information or clues, I'm out of ideas now, sorry. Anything I say now will be just guesswork shots in the dark I'm afraid. I like the idea of Walter Tross in your question comments. – Martin Nov 19 '15 at 23:39
@Serhiy I've just seen your question update code, I think it's your source code reader rather than the code itself..... maybe? – Martin Nov 19 '15 at 23:41

score 0 · Answer 2 · answered Nov 19 '15 at 22:30

0

The comment by Dean is actually very important. The HTML should have a <meta charset="UTF-8"> inside <head>.

That iconv call is actually not useful and, if you are right that you are getting your content as UTF-8, it is not necessary.

You should check the character set of your database connection. Your database can be encoded in UTF-8 but the connection could be in another character set.

Good luck!

answered Nov 19 '15 at 22:30

Braulio J. Solano

162
2
7

Browsers should detect UTF-8 without the meta, it's not a *necessary* requirement. I think the issue is with the database encoding. – Martin Nov 19 '15 at 22:34
Yeah, I felt like Dean's comment was a tangent. The file is bad, when I open it in a text editor, no browser needed. – Serhiy Nov 19 '15 at 22:43
1

@Serhiy I think it's your `b`inary marker on your fopen that is the issue. See my answer (at the bottom) – Martin Nov 19 '15 at 22:44
@Martin, thanks, I am reading through it, it's a bit much to take in at once, going through it, thanks for such details. – Serhiy Nov 19 '15 at 22:45

Stuck writing UTF-8 file via PHP's fwrite

2 Answers2

Edit: