Encoding issue. Where am I going wrong?

Question

I'm trying to pass data from my server and display it in a UWP Windows app. The data is stored in a mySQL database

This is being output via PHP to a web page here http://www.rwscripts.com/scorealerts/v3/request.php?action=getTeams using this code

// Serialize the data structure   
    $result = json_encode($data,JSON_PRETTY_PRINT);
    // Display the XML document   
    header('Content-type: application/json; charset=utf-8'); 

    print $result;

I'm then reading this in my app with HttpWebRequest and then deserializing the JSON with JSON.net

            JArray obj = JsonConvert.DeserializeObject(str.Trim()) as JArray;
            if (obj == null || obj.Count == 0) return;

            foreach (NotificationTeam nt in from JObject team in obj
                select
                    new NotificationTeam
                    {
                        Title = team.Value<string>("teamName"),
                        TeamID = team.Value<int>("tid"),
                        Followers = team.Value<int>("followers")
                    })
            {
                nt.Notifications = ScoreManager.GetMgr().GetTeamNotification(nt.TeamID);

                notificationTeams.Add(nt);
            }

the output when displayed in my app is like this

which part of the flow needs to be changed to display the unicode characters correctly?

Where would you suggest I use that? My web response comes back as a string `StreamReader requestReader = new StreamReader(response.GetResponseStream()); String webResponse = requestReader.ReadToEnd();` — Real World, May 20 '16 at 21:05
can you post your actual Unicode characters in text in question instead of picture? — techspider, May 20 '16 at 21:10
In the database it is 1. FC KÃƒÂ¶ln the C# output is "1. FC KÃ¶ln" and the json output can be seen in the link above — Real World, May 20 '16 at 21:14
you need to change the server connection with SET NAMES utf8 before inserting into Mysql — Mihai, May 20 '16 at 21:15
Can you post a full code including the HttpWebResponse reading part? — Kacper Ryniec, May 20 '16 at 21:20

score 5 · Accepted Answer · edited May 23 '17 at 11:58

There is no way you can fix this beyond the generated json - because it's completely wrong and here is why:

Special characters in team names (Köln) get stored as UTF-8 in your database. ö in UTF-8 is 0xc3 0xb6.
The output data however is then encoded (or just formatted) again as UTF-16 (aka Encoding.Unicode in C#) Here is where the trouble starts. ö in UTF-16 (and UTF-32) is 0x00 0xf6.
The UTF-8 character bytes get encoded as two separate UTF-16 characters \u00c3 and \u00b6 instead of just \u00f6. So instead of one utf-8 character, you end up with two utf-16 characters (that represent two bytes of the same utf-8 character).
Your app recognises the \u escape sequences and turns them (completly right from its point of view) into two separate UTF-16 characters (Ã¶).

Long story short, this is what happens to your strings:

ö in UTF-32 is f6000000
ö in UTF-16 is f600
ö in UTF-8 is 3c b6

Köln (Input)
K[0xc3][0xb6]ln (Sql UTF-8)
K\u00c3\u00b6ln (Json UTF-8 encoded as UTF-16)
KÃ¶ln (C# UTF-16 decoded)

Since json_encode expects a UTF-8 string, I suspect the problem occurs somewhere between the database and the encoding (php).

This post might give you a hint to where the encoding settings might be inconsistent:

UTF-8-all-the-way-through

In case you need to tinker with your settings, the output you will want is:

"teamName": "1. FC K\u00f6ln" or "teamName": "1. FC Köln" (should be fine too)

score 0 · Answer 2 · answered May 20 '16 at 21:01

0

Instead of this ..

$result = json_encode($data,JSON_PRETTY_PRINT);

.. maybe this .. ?

$result = json_encode($data,JSON_UNESCAPED_UNICODE);

.. or both.. ?

$result = json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE);

answered May 20 '16 at 21:01

Spooky

1,235
15
17

I tried this. It changes the html output but makes no difference to the app output – Real World May 20 '16 at 21:04
Well, I can't propose anything further, as I am not doing anything in C#. – Spooky May 20 '16 at 21:14

score 0 · Answer 3 · answered May 20 '16 at 21:13

0

I believe you need to get bytes of your Unicode characters and covert it to String.

var bytes = Encoding.Unicode.GetBytes(NotificationTeam.Title);
NotificationTeam.Title = Encoding.ASCII.GetString(bytes);

OR

new NotificationTeam
                    {
                        Title = Encoding.ASCII.GetString(Encoding.Unicode.GetBytes(team.Value<string>("teamName"))),
                        TeamID = team.Value<int>("tid"),
                        Followers = team.Value<int>("followers")
                    })

answered May 20 '16 at 21:13

techspider

3,370
13
37
61

Doing the latter I get the output "1\0.\0 \0F\0C\0 \0K\0?\0?\0l\0n\0" which is displayed as 1. FC K??ln in the app – Real World May 20 '16 at 21:19
what is your expected value? – techspider May 20 '16 at 21:19
try using https://dotnetfiddle.net/ to create your problem; execute below code there `using System; using System.Text; public class Program { public static void Main() { var unicode = "FC KÃƒÂ¶ln"; Console.WriteLine(Encoding.ASCII.GetString(Encoding.Unicode.GetBytes(unicode))); } }` – techspider May 20 '16 at 21:20
It should be 1. FC Köln – Real World May 20 '16 at 21:21
which language it is? – techspider May 20 '16 at 21:23
It's German. Using the json output `"1. FC K\u00c3\u00b6ln"` I also get ?? in dotnetfiddle.net using your code above – Real World May 20 '16 at 21:24
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/112547/discussion-between-real-world-and-techspider). – Real World May 20 '16 at 21:29

score 0 · Answer 4 · answered Dec 07 '19 at 03:39

0

This might be due to the bug in .NET Connector. In this case, you should specify

character_set_server=utf8mb4

in the config or [--character-set-server=utf8mb4][1] in the mysqld arguments

answered Dec 07 '19 at 03:39

druss

1,820
19
18

Encoding issue. Where am I going wrong?

4 Answers4