HttpResponse content with bad encoding

Question

There is some data I have to get from an official public API of a federal institute from Brazil. At first I was using JavaScript to get the data via HttpClient from the client-side and I was facing no issue.

Now, I decided to call this API from my own Asp.Net API, for security reason and to keep a buffer of the data instead of making a call of the federal API to each of my clients requests. But now, instead of a JSON formatted string, this is what I'm getting:

I know I'm not supposed to paste screenshots of texts, but I didn't find any other way to get this string. When I try to view this same string as text, this is what I get:

This is what I've been doing:

using (HttpClient httpClient = new HttpClient())
{
    var response = await httpClient.GetAsync("https://servicodados.ibge.gov.br/api/v1/localidades/municipios");
    if (response?.IsSuccessStatusCode ?? false)
    {
        var str = await response.Content.ReadAsStringAsync();
        ...
    }
}

I've tried other reading methods, although I know all of them are supposed to give the same answer. I've tried reading the data as byte array and tried multiple Encoding: ASCII, Unicode, UTF8...

var array = await httpClient.GetByteArrayAsync("https://servicodados.ibge.gov.br/api/v1/localidades/municipios");
var str = Encoding.UTF8.GetString(array);

No success.

By simply pasting this very same URI on my browser, I get the result I want which is just a huge JSON.

This is the API documentation, in Brazilian Portuguese.

score 0 · Accepted Answer · answered Mar 27 '20 at 18:52

Turns out the response was compressed in a gzip. I had to decompress it first,

This was the solution to my problem:

using (HttpClient httpClient = new HttpClient())
{
    var bytes = await httpClient.GetByteArrayAsync("https://servicodados.ibge.gov.br/api/v1/localidades/municipios");
    using (var msi = new MemoryStream(bytes))
    using (var mso = new MemoryStream())
    {
        using (var gs = new GZipStream(msi, CompressionMode.Decompress))
        {
            gs.CopyTo(mso);
        }

        var response = Encoding.UTF8.GetString(mso.ToArray());
    }
}

Source

score 0 · Answer 2 · answered Jun 02 '23 at 19:46

Yes, it is due to gzip compression of data by server, not wrong Encoding!

A better way to unzip the compressed data automatically, is to use a HttpClientHandler or SocketsHttpHandler with proper AutomaticDecompression when you create your HttpClient as in following code:

var httpHandler = new SocketsHttpHandler() {
      AutomaticDecompression = DecompressionMethods.GZip
                             | DecompressionMethods.Deflate
};

var httpClient = new HttpClient(httpHandler);

This way the output of ReadAsStringAsync or other reading methods will be automatically decoded data as you want.

HttpResponse content with bad encoding

2 Answers2