Why am I getting encrypted data back from an API call?

Question

I'm trying to log into a rest api using the below code and though fiddler can decode the response, it's not being decoded by the WebClient. I am using a class that extends WebClient which I found on SO that allows me to access the response. It is also below.

        using (WebClientWithResponse client = new WebClientWithResponse())
        {
            client.Headers.Add("cache-control", "no-cache");
            client.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
            client.Headers.Add("Authorization", basicAuth);
            client.Headers.Add("Accept", "application/json, text/json, text/x-json, text/javascript, application/xml, text/xml");
            client.Headers.Add("User-Agent", "MyAppName/21.1.1");
            client.Headers.Add("Host", "login.company.com");
            client.Headers.Add("Accept-Encoding", "gzip, deflate");

            using (StreamWriter sw = new StreamWriter(client.OpenWrite("https://login.companyapi.com/services/oauth2/token", "POST")))
            {
                sw.Write(loginContent);
            }

            //string response = Encoding.UTF8.GetString(client.Response);
            string response = Encoding.UTF8.GetString(Decompress(client.Response));
            File.WriteAllText(".\\response.txt", response);

            Console.WriteLine(response);
        }

public class WebClientWithResponse : WebClient
{
    // we will store the response here. We could store it elsewhere if needed.
    // This presumes the response is not a huge array...
    public byte[] Response { get; private set; }

    protected override WebResponse GetWebResponse(WebRequest request)
    {
        var response = base.GetWebResponse(request);
        var httpResponse = response as HttpWebResponse;
        if (httpResponse != null)
        {
            using (var stream = httpResponse.GetResponseStream())
            {
                using (var ms = new MemoryStream())
                {
                    stream.CopyTo(ms);
                    Response = ms.ToArray();
                }
            }
        }
        return response;
    }
}

So below you can see what Fiddler shows (after I click the decode button) and what I receive in my app.

This is how Notepad++ looks when loading the saved response file.

� ]�Ks�@�K8�!@Ъ�>P@M�B��H@��ݕ���C�9~��3��JA�T�0��!�+��{�gȟu��^r��1��(YtU+��H��4JWs��";d�E�[]�4��9o�,� �G�Ǘ@��뺧=j��0 �IT��Jd�0E](R��.Ir��͒�V</؝�(2�Jߏ��!�1��q�S=U�4� �턔'�R��"ײmϱ��,rF�I��T�Lp��t�لM�3�:��(�[g�:�x��}}*��p

I've never had this issue before and can't for the life of me figure out why.

EDIT: Adding the response headers as requested by Marc Gravell

HTTP/1.1 200 OK 
Date: Fri, 16 Apr 2021 19:33:23 GMT 
Strict-Transport-Security: max-age=31536000; includeSubDomains 
X-Content-Type-Options: nosniff 
X-XSS-Protection: 1; mode=block 
Cache-Control: no-cache,must-revalidate,max-age=0,no-store,private 
Set-Cookie: BrowserId=m6jRGp7qEeuWJ50fQL0HHA; domain=.company.com; path=/; expires=Sat, 16-Apr-2022 19:33:23 GMT; Max-Age=31536000 
Expires: Thu, 01 Jan 1970 00:00:00 GMT 
X-ReadOnlyMode: false 
Content-Type: application/json;charset=UTF-8 
Vary: Accept-Encoding 
Content-Length: 368

EDIT2: Marc Gravell gave me the idea that it wasn't decompressed. That turned out to be correct. I added this function found here to decompress the response.

    public static byte[] Decompress(byte[] data)
    {
        using (var compressedStream = new MemoryStream(data))
        using (var zipStream = new GZipStream(compressedStream, CompressionMode.Decompress))
        using (var resultStream = new MemoryStream())
        {
            zipStream.CopyTo(resultStream);
            return resultStream.ToArray();
        }
    }

What is the content-type, content-encoding, etc - in fact, ALL of the response headers? Most likely it either hasn't applied decompression, or the data **isn't text** — Marc Gravell, Apr 16 '21 at 21:07
@MarcGravell Thanks! It wasn't decompressed. Make an answer and I'll mark it accepted. — tolsen64, Apr 16 '21 at 21:45
It looks like they missed off the encoding header to tell you that, no? So the client didn't know to decompress it? — Marc Gravell, Apr 16 '21 at 21:48
Why are you tweaking WebClient instead of using HttpWebRequest / HttpClient (the latter, possibly)? Decompression is automatic. So is the detection of the Encoding, which is instead set to `Encoding.Default` in WebClient (unless otherwise specified - as you would know *in advance*; still no auto-detection). — Jimi, Apr 16 '21 at 21:54
@Jimi the decompression is automatic if the server includes the correct headers; it did not — Marc Gravell, Apr 16 '21 at 21:58
@Marc Gravell If the Stream is a GzipStream, it's auto-detected anyway. Unless the other side is sending a pre-compressed array of bytes. Who does that? -- But, I cannot test it; the OP should. Setting `AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate` beforehand, I mean, since the default is `DecompressionMethods.None`. — Jimi, Apr 16 '21 at 22:19

score 2 · Accepted Answer · answered Apr 16 '21 at 22:00

It looks like the server you are connecting to applied gzip compression, but failed to include the Content-Encoding response header (which they are meant to include), so the client had no way of knowing that the data was compressed, and didn't decompress it for you automatically. Honestly, you should report this to the server owner as a protocol failure, but for now: you could manually decompress the data using GZipStream. Note that if they fix this, you may need to tweak your code again.

Why am I getting encrypted data back from an API call?

1 Answers1