1

I have about a million urls pointing to HTML pages on a public web server that I want to save to my disk. Each of these is about the same size, ~30 kilobytes. My url lists are split about evenly in 20 folders on disk, so for simplicity I create one Task per folder, and in each task I download one URL after the other, sequentially. So that gives me about 20 parallel requests at any time. I'm on a relatively crappy DSL, 5mbps connection.

This represents several gigabytes of data so I'm expecting the process to take several hours, but I'm wondering if I could make the approach any more efficient. Is it likely I'm making the most out of my connection? How can I measure that? Is 20 parallel downloads a good number or should I dial up or down?

The language is F#, I'm using WebClient.DownloadFile for every url, one WebClient per task.

==================================

EDIT: One thing that made a huge difference was adding a certain header to the request:

let webClient = new WebClient()
webClient.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")

This cut the size of downloads from about 32k to 9k, resulting in enormous speed gains and disk space savings. Thanks to TerryE for mentioning it!

Asik
  • 21,506
  • 6
  • 72
  • 131
  • By default .net has a connection limit of 2 per server. You'll need to change that if you want more parallel downloads. – CodesInChaos Jul 25 '12 at 20:59

2 Answers2

2

If you are using a downloader API, then make sure that it is issuing a

Accept-Encoding: gzip, deflate

request header so that the site that you are scraping knows to return compressed HTML. (Most webservers wil be configured to compress the HTML data streams if the client uses this request header to let the server know that it will accept compressed data streams.)

This will reduce the data transferred by roughly a factor of 4. (E.g. this page was 40K raw HTML, but only 10K was transferred to my browser (the HTML is zipped).

TerryE
  • 10,724
  • 5
  • 26
  • 48
1

I would only parallelize until reaching the limit of the connection speed. If each single request is saturating your DSL connection, running them in parallel won't gain you anything, and might get you blocked.

First measure your capacity with a tool like http://wowrack.speedtest.net. Then parallelize until your throughput reaches this value. There are a variety of ways to monitor your current network usage, the easiest is to go to the Windows Task Manager and hit the Network tab.

Also make sure you're keeping a connection open to the server, and not reopening it for every request. This will cause unnecessary overhead.

Joseph Sheedy
  • 6,296
  • 4
  • 30
  • 31
  • Windows Task Manager only gives me the percentage of traffic on the network connection (100Mbps), it's a tiny number of course. Also do you have any idea if WebClient.DownloadFile opens a new connection every time? – Asik Jul 25 '12 at 21:16
  • 1
    You could try a tool like http://codebox.org.uk/pages/bitmeter2 to measure your throughput. I'm not sure if the WebClient.DownloadFile method keeps the connection open in your environment, but you might want to read through http://www.pcreview.co.uk/forums/does-webclient-keep-connection-open-t3421233.html – Joseph Sheedy Jul 25 '12 at 21:33
  • Thanks, bit meter OS works really well. It looks like I'm saturating the connection, so my current approach seems to be good. – Asik Jul 26 '12 at 02:34