Downloading JSON with WebClient results in weird unicode-like characters?

Question

So I can make a request in my browser to, https://search.snapchat.com/lookupStory?id=itsmaxwyatt

and it will give me back JSON, but if I do it via web client, it seems to give me back a very obfuscated string? I can provide it all, but have truncated for now:

�x��ƽ��o�Cj�_��˗��89:�/�[��/� h��#l��ٗC��U.�gH�,��qOv�_� �_��σҭ

So, here is the Csharp code:

using var webClient = new WebClient();
webClient.Headers.Add ("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:89.0) Gecko/20100101 Firefox/89.0");
webClient.Headers.Add("Host", "search.snapchat.com");
webClient.DownloadString("https://search.snapchat.com/lookupStory?id=itsmaxwyatt")

I have also tried in a http rest client without any headers, and it still returns JSON.

Tried with encoding:

using var webClient = new WebClient();
webClient.Headers[HttpRequestHeader.AcceptEncoding] = "gzip";
webClient.Encoding = Encoding.UTF8;
webClient.Headers.Add ("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:89.0) Gecko/20100101 Firefox/89.0");
webClient.Headers.Add("Host", "search.snapchat.com");
Console.WriteLine(Encoding.UTF8.GetString(webClient.DownloadData("https://search.snapchat.com/lookupStory?id=itsmaxwyatt")));

Does this answer your question? [Automatically decompress gzip response via WebClient.DownloadData](https://stackoverflow.com/questions/2973208/automatically-decompress-gzip-response-via-webclient-downloaddata) — Progman, Jun 16 '21 at 17:24
so I have updated my question with a code example using this approach, but still very cryptic response. — async life, Jun 16 '21 at 17:26

score 1 · Accepted Answer · answered Jun 16 '21 at 17:38

Following @Progman comment, all you need is to do the following:

// You can define other methods, fields, classes and namespaces here
class MyWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
        request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
        return request;
    }
}
void Main()
{
    using var webClient = new MyWebClient();
    webClient.Headers.Add("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:89.0) Gecko/20100101 Firefox/89.0");
    webClient.Headers.Add("Host", "search.snapchat.com");
    var str = webClient.DownloadString("https://search.snapchat.com/lookupStory?id=itsmaxwyatt");
    Debug.WriteLine(str);
}

Downloading JSON with WebClient results in weird unicode-like characters?

1 Answers1