Error with utf8 characters and htmlspecialchars in yii

Question

I have a problem with the char encoding in yii. If I create a new webapp:

 ./Yii-framework/framework/yiic webapp MyTest

Then go to /protected/views/layouts/main.php and change the footer to a text with an utf8 character, for example

<div id="footer">
        Cópyrîgth <br />
</div>

Refresh the page and everything is ok. Nice! ;)

And then I try to log in with an utf8 character in the username, for example ádmin, it crashes saying:

Error 500

htmlspecialchars(): Invalid multibyte sequence in argument

So I checked this article about unicode in yii

and then I went to /protected/config/main.php and added this line at the start:

header('Content-Type: text/html; charset=utf-8');

Retrying the same login again it works (doesn't crash) but now the footer is broken and shows:

C�pyr�ght

I've tried other combinations like explained in the "Unicode in yii" article but none of them make both things work at the same time.

Any ideas for solving this problem?

Note: I can't change to the php.ini file.

I also tried the AddDefaultCharset UTF-8 option in the .htaccess file and put it in the folder at the /MyTest/ is that the correct folder referred in the article as: your DocumentRoot ?

Thanks

those question marks are the result of invalid UTF-8 byte sequences. Most probably you were using an editor that saved text in a single-byte encoding, like, e.g., [ISO 8859-1](http://en.wikipedia.org/wiki/ISO/IEC_8859-1). In all single-byte encodings that are extensions of ASCII, the extended part has byte values >= 128. All single-byte characters of UTF-8 are < 128, all multi-byte characters consist of bytes >= 128. This is why ISO 8859-x characters with diacritics become question marks: they could never be valid UTF-8, except in quite unlikely combinations. — Walter Tross, Mar 20 '13 at 15:58

score 6 · Accepted Answer · answered Jun 11 '12 at 22:12

6

I'm not at all familiar with yii, but, if you want to paste literal unicode characters into a file, you need to make sure that your text editor saves the file using a unicode encoding, such as utf8. Try utf8, without a BOM.

My experience is that text editors behave strange when you change the encoding setting and there's already encoded characters in it. Just start over with a fresh file, change the encoding, then paste the characters in.

answered Jun 11 '12 at 22:12

goat

31,486
7
73
96

"using a unicode encoding, such as utf8" is not a good advice, it should be "using the UTF-8 Unicode encoding". Other [Unicode](http://en.wikipedia.org/wiki/Unicode) encodings are not recommended for the web. And the [BOM](http://en.wikipedia.org/wiki/Byte_order_mark) (byte order mark) makes little to no sense in UTF-8, most editors don't even consider this option for UTF-8 (correctly). – Walter Tross Mar 24 '13 at 10:03
I have to correct myself: apparently there is even an example of an editor that is **not** able to save an UTF-8 encoded file **without** a BOM: Microsoft's Notepad. Windows users not wanting to use an IDE's editor can still use Notepad++, which is free and very user friendly. – Walter Tross Mar 24 '13 at 13:46

Walter Tross · Answer 2 · 2013-03-23T20:42:56.470

First off, you need to understand that a character with a diacritic like ó or î (from your example) is not automatically a "utf-8 character". It is simply a character that has different encodings (if any) in different character sets, even in those character sets that have the basic single-byte ASCII part in common (i.e., the English alphabet, the digits, the most common punctuation, and a few more). You could call it a "problematic character", but not a "utf-8 character".

So, when you wrote your footer <div>, you did NOT write it UTF-8 encoded. Your editor saved those characters in a single-byte encoding, like ISO 8859-1 or one of its relatives.

Browsers normally automatically detect the encoding used in a page, if it is not specified. This is why you were initially able to see in the browser exactly what you had written in your editor.

Then you tried to log in with a "problematic character" in the username. The browser had interpreted your page as having a single-byte encoding, so this caused it to encode your form input the same way, and send it single-byte-encoded back to the server. The PHP code had not been written with this possibility in mind, apparently, because it did not correctly set the third parameter of htmlspecialchars(), which is "UTF-8" by default (starting from PHP 5.4.0 - it was "ISO-8859-1" before). Since a single-byte encoded string with "problematic characters" almost never is a valid UTF-8 string (see my comment to your question, it's the second comment), htmlspecialchars() rejected it.

Then you correctly added the header('Content-Type: text/html; charset=utf-8');, which disabled the automatic charset detection by the browser. At this point it became evident that your file with the footer <div> was not UTF-8 encoded (see again my comment for the explanation of the question marks that appear instead of the "problematic characters").

So all you are left to do is convince your editor to save files UTF-8 encoded. As others have noted, saving the file in a different encoding does not work in all editors. Starting from a fresh file is sometimes the solution, maybe after having set the default encoding of your editor to UTF-8.

To check the encoding, you can use the file command in a shell. Its output should be something like

main.php: PHP script, UTF-8 Unicode text

Or else, you could use the od -tx1z command, which dumps your file (maybe | less), as a sequence of hex bytes with the corresponding string on the side. If the file is single-byte encoded, your "problematic characters" will be single bytes >= 0x80. If it is UTF-8 encoded, they will be sequences of 2 bytes (others will be 3 or more bytes), all >= 0x80, while the "non-problematic characters" will continue to be single bytes < 0x80.

The article you mention seems to be well-written, just follow it.

You don't need the AddDefaultCharset directive in the .htaccess file, though, if all your pages are generated with the Content-Type: text/html; charset=utf-8 HTTP header, because the effect of the Apache directive is exactly the same (and it is good to keep the control on encoding inside PHP).

Adding the <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> has the same effect, for the browser, as the above HTTP header (note the http-equiv). The HTTP header is cleaner, but this additional meta tag may help in case a page is saved without the header's information.

Most importantly, don't be afraid of UTF-8, because it is your friend!

(...but, from the answer that got your bounty, I see that you, like many people, continue to think that understanding character encodings is too difficult for you ☹)

+10 for comprehensive answer and sad face at end which is in utf — , Jul 26 '13 at 11:57

score 2 · Answer 3 · answered Jun 13 '12 at 13:15

2

First, you should remove the header call at the main.php file, it might create problems for you in the future.

Second, I would do what rambo coder suggested and make sure that your files are saved as UTF8 in your editor.

answered Jun 13 '12 at 13:15

subdee

136
2

score 1 · Answer 4 · answered Mar 15 '13 at 20:31

The above solutions seems to be the right way since Yii doesnt really have a problem with unicodes but you could also perform some additional checks like the charset within the meta tag in your Html page is set to utf-8 and instead of writing plain html you can use Chtml::encode(Copyright) so that yii would handle the encoding. For the username part make sure the Default Charset in your database is also set to utf8.

score 1 · Answer 5 · edited May 23 '17 at 10:29

1

For htmlspecialchars problem check this answer: https://stackoverflow.com/a/3803972/133408

You have to specify encoding as 3rd parameter of htmlspecialchars

edited May 23 '17 at 10:29

Community

1
1

answered Mar 20 '13 at 16:31

score 0 · Answer 6 · answered Mar 21 '13 at 05:58

0

The best way around this is to use http://www.utexas.edu/learn/html/spchar.html - in your case Cópyrîght would appear as Cópyrîght

Also, I'll add in the HTML <meta charset="utf-8"> to make sure browsers are behaving themselves.

answered Mar 21 '13 at 05:58

Alvin K.

4,329
20
25

1

Doubtly best way, in properly setup utf system there is no need to encode characters into html entities. – Mar 21 '13 at 07:43
Doesnt deserve a downvote, because although not technically right, this is an option I am glad to know about. – rockstardev Mar 21 '13 at 16:27
"best way AROUND this" means to bypass all UTF8 complication, esp. true when working with multi-byte characters and seeing � showing up for user-agent (browsers) with different font-set. – Alvin K. Mar 21 '13 at 17:41

score 0 · Answer 7 · edited May 23 '17 at 12:02

I had this problem too - specifically when I was trying to display utf text from db. I changed all the colations and types in mysql to utf8-bin - but still no love... then I tried to chnage all of my layouts and views with the meta tags etc... hell, I even looked at Japanese websites source code and pasted that stuff in... NOTHING WORKED _ ... UNTIL... I came across THIS post: Yii And UTF8 Display, UTF8 works with mysqli but not yii backend Turns out, you need to tweek a setting in my main.php in the config file, under components.. f

Error with utf8 characters and htmlspecialchars in yii

7 Answers7

Linked