0

Im trying to download strings (xml) from multiple sources asynchronously and wait for all to complete.

But when using Task.Factory.StartNewto create multiple tasks, they all seem to download synchronously, one after another. All strings combined should weight aprox. 200kb so it shouldnt be a problem for my internet connection that can download 640kb per second.

Here is the code I'm using:

var taskList = new List<Task>();

foreach (var feed in feeds)
    taskList.Add(Task.Factory.StartNew(() => DownloadFeed(feed)));

Task.WaitAll(taskList.ToArray());

Method that downloads:

void DownloadFeed(string url)
{
    using (Webclient client = new WebClient())
      concurrentBag.Add(client.DownloadString(url));
}

WebClient's async methods do work, they download asynchronously reducing the total downloading time to just a fraction. And while that should be good enough for my project, I'm still curious what is wrong with my code that uses Tasks?

DumbCode
  • 315
  • 2
  • 7
  • 3
    "They all **seem**", how are you observing this sequential behavior? – Lasse V. Karlsen May 26 '19 at 20:59
  • @LasseVågsætherKarlsen I ran some tests and the code using Tasks takes virtually the same time as synchronous code. So my conclusion was this code is not downloading asynchronously – DumbCode May 26 '19 at 21:04
  • Duplicate of https://stackoverflow.com/questions/866350/how-can-i-programmatically-remove-the-2-connection-limit-in-webclient – Wiktor Zychla May 26 '19 at 21:07
  • It is not *downloading* asynchronously but it should be *executing* in parallel, since you've forcibly created individual tasks for this. They should all be executing using the the threadpool, but the threadpool has a limit. If you're downloading 1000 links like this, only a number of threads will be used for this, usually related to the number of cores your computer. A true async download, however, might spin up many many more downloads. – Lasse V. Karlsen May 26 '19 at 21:08
  • 4
    Why don't you use the async methods on `WebClient` instead of trying to roll your own? https://learn.microsoft.com/en-us/dotnet/api/system.net.webclient.downloadstringasync This might allow the underlying system to offload the asynchronous nature to I/O rather than to CPU threads. – David May 26 '19 at 21:10
  • And then.. After the fact the Task Scheduler will be limiting your Threads, The website is most likely throttling you. – TheGeneral May 26 '19 at 21:18
  • @TheGeneral I should have mentioned they are all different sources that request connections to completely different servers – DumbCode May 26 '19 at 21:20
  • @LasseVågsætherKarlsen Correct, but my question was why are they not downloading asynchronously? Please note that I'm downloading only 4 at a time. There has to be a bottleneck somewhere, a limitation. – DumbCode May 26 '19 at 21:25
  • 1
    Yes? As I said, the threadpool. Hang on, are you measuring the time it takes to do *4* downloads? Likely overhead all over the place is killing this. Please do some real measurements. – Lasse V. Karlsen May 26 '19 at 21:31
  • I don't think you are at the stage of asking a question about this, tasks and `WhenAll` do run in parallel until you hit task scheduler limit on the thread pool, your methods for testing for parallel aren't suitable, you would get more mileage out of start finish debug logs, you should be use in the async and await pattern anyway. At this stage there is not really any proof that this is not working, just assumptions – TheGeneral May 26 '19 at 23:03

0 Answers0