MySQL to PHP to NSData to NSString - Where has my UTF8 encoding gone wrong?

Question

I'm trying to write a simple PHP web service as a bridge between a MySQL database on my web server and an Objective-C Cocoa desktop app. Every link in the chain is working in UTF-8, as best as I can tell, but I can't seem to get the non-standard characters to display in Cocoa; instead, my interface elements show only the escapes.

I'm open to any solution that lets me fix this problem at any stage; I'm writing the PHP and the Obj-C side, so I should be able to deal with problems on the server side or on the client side. I just don't know which mole to whack at this point.

The Database

The information is coming from a table in a MySQL database; the database collation and each field are set to utf_general_ci.

PDO

My PHP web service fetches the data using PDO:

    $pdo = new PDO('mysql:host=mysql.hostname.com;dbname=tst_data_model', 'user', 'password', array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
if ($pdo) {} else {die();}

$stmt = $pdo->prepare("SELECT * FROM team");
$stmt->execute();

PHP

I then echo the data:

header('Content-Type: text/html; charset=utf-8'); 
echo("<TEAM>\r");
echo("<team_id>{$row['team_id']}</team_id>\r");
echo("<name_full_native>{$row['name_full_native']}</name_full_native>\r");
echo("<name_short_native>{$row['name_short_native']}</name_short_native>\r");
echo("<name_full_english>{$row['name_full_english']}</name_full_english>\r");
echo("<name_short_english>{$row['name_short_english']}</name_short_english>\r");
echo("</TEAM>\r\r");

NSDATA

The desktop app fetches this data using NSURLConnection:

NSData* returnValue = [NSURLConnection sendSynchronousRequest:teamArrayRequest returningResponse:NULL error:NULL];

NSString

Finally, I pull the string out of the data, specifying UTF8 string encoding:

NSString* rVString = [[NSString alloc] initWithData:returnValue encoding:NSUTF8StringEncoding];

When I view this string in the debugger, or send it to an NSTextView, all I see are \u escapes:

Borussia M\U00f6nchengladbach

Quer\U00e9taro

Real Espa\U00f1a

Am I missing something? Where are these escapes coming from?

I suspect that the problems are with MySQL or PHP. Cocoa has had solid Unicode support since day 1, but PHP is blind to encodings and MySQL's support for UTF-8 was so terrible it was embarrassing, until recently. — Dietrich Epp, Oct 17 '12 at 04:36
It can also be your system. If you connect to the server and execute the query "by hand" on mysql: Does your screen show the correct characters? Check the table's encoding using this link: http://stackoverflow.com/questions/6103595/how-to-check-if-mysql-table-is-utf-8-and-has-storageengine-innodb — fiacobelli, Oct 17 '12 at 04:40
Never mind; solved it. I'm not allowed to self-answer for another seven hours, so I'll post the detailed answer in the morning. The short version: I'm an idiot. The slightly longer version: The string was in UTF-8, but I wasn't looking at the string; I was looking at the output of the -description method of the NSArray in which the string was stored as soon as I fetched it. — chapka, Oct 17 '12 at 04:57

score 0 · Accepted Answer · edited Jun 20 '20 at 09:12

Uh...never mind. Turned out this was a case of PEBCAK.

In fact, everything was in UTF8 after all, all the way to the NSString. What confused me is that what I was viewing in the debugger, and passing to the NSTextView, wasn't the NSString; it was the output of the -description method of either an NSArray or an NSDictionary containing the string that I was immediately assigning it to after I'd fetched it.

Turns out the -description method of either NSArray or NSDictionary always, for some reason, outputs UTF-8 characters other than standard ASCII characters as /u escapes.

When I actually pulled the string out of the array of dictionaries I was using to parse and store it and plugged the string directly into the NSTextView, it displayed as perfect UTF8:

Borussia Mönchengladbach

Querétaro

Real España

So...that's the answer. [NSArray description] and [NSDictionary description] do not display every character in a string value as it will display as an NSString. I guess this is a choice the coders made so that whitespace characters, etc. don't show up in plist descriptions.

Although I know I've seen Chinese characters in plist descriptions and debugger output before...so it's still a little mysterious. Still, problem solved. Sorry to waste the hive-mind's time, and I hope the explanation helps someone down the road.

MySQL to PHP to NSData to NSString - Where has my UTF8 encoding gone wrong?

1 Answers1