C# WebClient DownloadString returns gibberish

Question

I am attempting to view the source of http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/ using the code:

String URL = "http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/";

WebClient webClient = new WebClient();

webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows; Windows NT 5.1; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4");
webClient.Encoding = Encoding.GetEncoding("Windows-1255");

string download = webClient.DownloadString(URL);

webClient.Dispose();

Console.WriteLine(download);

When I run this, the console returns a bunch of nonsense that looks like it's been decoded incorrectly.

I've also attempted adding headers with no avail:

webClient.Headers.Add("user-agent", "Mozilla/5.0 (Windows; Windows NT 5.1; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4");    
webClient.Headers.Add("Accept-Encoding", "gzip,deflate");

Other websites all returned the proper html source. I can also view the page's source through Chrome. What's going on here?

Response is gzipped, you should decompress it, see http://stackoverflow.com/questions/2973208/automatically-decompress-gzip-response-via-webclient-downloaddata — Antonio Bakula, May 28 '12 at 01:40

Antonio Bakula · Accepted Answer · 2012-05-28T02:03:42.363

4

Response of that URL is gzipped, you should decompress it or set empty Accept-Encoding header, you don't need that user-agent field.

  String URL = "http://simpledesktops.com/browse/desktops/2012/may/17/where-the-wild-things-are/";    
  WebClient webClient = new WebClient();    
  webClient.Headers.Add("Accept-Encoding", "");
  string download = webClient.DownloadString(URL);

edited May 28 '12 at 02:03

answered May 28 '12 at 01:54

Antonio Bakula

20,445
6
75
102

score 1 · Answer 2 · answered Nov 11 '13 at 13:32

I've had the same thing bug me today.

Using a WebClient object to check whether a URL is returning something.

But my experience is different. I tried removing the Accept-Encoding, basically using the code @Antonio Bakula gave in his answer. But I kept getting the same error every time (InvalidOperationException)

So this did not work:

WebClient wc = new WebClient();
wc.Headers.Add("Accept-Encoding", "");
string result = wc.DownloadString(url);

But adding 'any' text as a User Agent instead did do the trick. This worked fine:

WebClient wc = new WebClient();
wc.Headers.Add(HttpRequestHeader.UserAgent, "My User Agent String");
System.IO.Stream stream = wc.OpenRead(url);

Your mileage may vary obviously, also of note. I'm using ASP.NET 4.0.30319.

C# WebClient DownloadString returns gibberish

2 Answers2