C# avoid bad encoding using HttpRequestMessage

Question

I'm getting a response from an API call and I'm using the HttpRequestMessage to set up my "get" request and then HttpClient to stream the response and return a string. However, within the response, I'm getting \u2019 instead of ' and in when I convert this result into excel (using JsonConvert and CsvWriter), I'm getting â€™ instead of ' in my csv. Do I miss something at the headers level when requesting the API's response ?

public static string GetResponse_CFRA(string oauth2_token, string apiKey, string uri)
        {
            var httpRequestMessage = new HttpRequestMessage
            {
                Method = HttpMethod.Get,
                RequestUri = new Uri(uri),
                Headers = {
                    { "Authorization", $"Bearer {oauth2_token}"},
                    { "x-api-key", apiKey}
                }
            };

            // Get the response from the API
            using (var client = new HttpClient())
            {
                try
                {
                    var response = client.SendAsync(httpRequestMessage).Result;
                    HttpContent responseContent = response.Content;
                    var responsedata = responseContent.ReadAsStringAsync();
                    string data = responsedata.Result;
                    return data;
                }

                catch
                {
                    string sorry = "Please call the admin";
                    return sorry;
                }
            }
        }

The characters \u2019 and â€™ are Unicode characters that represent the single quote ('). It seems that the API you're calling is returning a response that has these characters encoded in Unicode, but when you convert the response to a string, the character encoding is not being properly handled and the characters are being displayed incorrectly. — Kozydot, Dec 23 '22 at 11:02

canton7 · Accepted Answer · 2022-12-23T14:16:14.623

\u2019 (the Unicode character U+2019) is the right single quotation mark ’, i.e. the slightly curved version of '.

When encoded as UTF-8 (which is the encoding which .NET uses by default to write files) it's represented by the byte sequence 0xE2 0x80 0x99.

However, if you take the bytes 0xE2 0x80 0x99 and interpret them not as UTF-8 but rather using the Windows 1252 code page (which is one of the default single-byte code pages on Windows, depending on your locale), 0xE2 maps to â, 0x80 to € and 0x99 to ™.

So your problem is that you've got a text file which uses UTF-8 to encode characters to bytes, but Excel is trying to read it using Windows-1252 instead, which maps different characters onto those bytes.

Tell Excel to interpret the CSV file as UTF-8, add a UTF-8 byte order mark (BOM, which is the sequence 0xEF 0xBB 0xBF) to the start of the file, or change the encoding from UTF-8 to Windows-1252 when you save the file.

To write the string to a file using UTF-8 with BOM you'll need to manually specify the encoding as Encoding.UTF8, e.g.:

File.WriteAllText(path, contents, Encoding.UTF8);

Indeed you're right ! So I actually used the Encoding.UTF8 at the moment I write my csv using this line: `code` using (var writer = new StreamWriter("C:/My_Path/sample.csv", false, Encoding.UTF8))Encoding.UTF8 Thank you so much ! — Pierre-Emmanuel giuglaris, Dec 23 '22 at 13:37

C# avoid bad encoding using HttpRequestMessage

1 Answers1