1

I am using System.Net.WebClient.DownloadFile to download a large number of html files from a public web server. I would like to enable compression because that could cut the data transfers by a large factor. I was told in a previous question that I should set the "Accept-Encoding: gzip, deflate" header. WebClient has a Headers property that would allow doing that.

I have a few questions about this:

  1. Assuming the web server recognizes the header, would this work with the DownloadFile method, or only with the DownloadData method? I couldn't find any information about this on MSDN or other sites.
  2. If it doesn't work with DownloadFile, what would be the simplest way of implementing downloading compressed data to a file?
  3. Does any other header need to be set for the server to compress its data? In an old codinghorror.com entry, Jeff Atwood said the "UserAgent" header also needs to be set, but I don't know what the value should be.
Community
  • 1
  • 1
Asik
  • 21,506
  • 6
  • 72
  • 131

1 Answers1

2
  1. DownloadFile is just a wrapper around a raw WebRequest; it just processes the result and packages it as a "file" or a byte array.

  2. It works; but the data that is sent to the local file is still compressed, you'll have to decompress it manually.

  3. You get a default UserAgent. There isn't any particular UserAgent values necessary, unless the site your accessing requires one. But, you'll have to find that out. (Jeff's post suggest Google requires one; but I don't know if that still true)

Peter Ritchie
  • 35,463
  • 9
  • 80
  • 98