C# How to process several web requests at once

Question

I have been reading a lot about ThreadPools, Tasks, and Threads. After awhile I got pretty confused with the whole thing. Lots of people saying negative/positive things about each... Maybe someone can help me find a solution for my problem. I created a simple diagram here to get my point across better.

enter image description here

Basically on the left is a list of 5 strings (URL's) that need to be processed. In the center is just my idea of a handler that has 2 events to track progress. Inside that handler it takes all 5 URL's creates separate tasks for them, shown in blue. Once each one complete I want each one to return the webpage results to the handler. When they have all returned a value I want the OnComplete to be called and all this information passed back to the main thread.

Hopefully you can understand what I am trying to do. Thanks in advance for anyone who would like to help!

Update I have taken your suggestions and put them to use. But I still have a few questions. Here is the code I have built, mind it is not build proof, just a concept to see if I'm going in the right direction. Please read the comments, I had included my questions on how to proceed in there. Thank you for all who took interest in my question so far.

public List<String> ProcessList (string[] URLs)
{
    List<string> data = new List<string>();
    for(int i = 0; i < URLs.Length - 1; i++)
    {
        //not sure how to do this now??
        //I want only 10 HttpWebRequest running at once.
        //Also I want this method to block until all the URL data has been returned.            
    }
    return data;
}

private async Task<string> GetURLData(string URL)
{
    //First setup out web client
    HttpWebRequest Request = GetWebRequest(URL);
    //
    //Check if the client holds a value. (There were no errors)
    if (Request != null)
    {
        //GetCouponsAsync will return to the calling function and resumes
        //here when GetResponse is complete.
        WebResponse Response = await Request.GetResponseAsync();
        //
        //Setup our Stream to read the reply
        Stream ResponseStream = Response.GetResponseStream();

        //return the reply string here...
    }
}

Is a simple Parallel.ForEach out of the question? http://msdn.microsoft.com/en-us/library/dd460720(v=vs.110).aspx -- Ah, I see you want only 10 at a time. Take a look at this SO question for limiting: http://stackoverflow.com/questions/9290498/how-can-i-limit-parallel-foreach — Erik Noren, Dec 11 '13 at 22:48
So if I use a Parallel.ForEach I won't need to have the async Task ? I can just have private string GetURLData(string URL) ? — user2868614, Dec 11 '13 at 23:10

score 0 · Answer 1 · answered Dec 11 '13 at 06:36

0

As @fendorio and @ps2goat pointed out async await is perfect for your scenario. Here is another msdn article

http://msdn.microsoft.com/en-us/library/hh300224.aspx

answered Dec 11 '13 at 06:36

Dilish

429
4
11

score 0 · Answer 2 · answered Dec 11 '13 at 06:42

It seems to me that you are trying to replicate a webserver within a webserver.

Each web request starts its own thread in a webserver. As these requests can originate from anywhere that has access to the server, nothing but the server itself has access or the ability to manage them (in a clean way).

If you would like to handle requests and keep track of them like I believe you are asking, AJAX requests would be the best way to do this. This way you can leave the server to manage the threads and requests as it does best, but you can manage their progress and monitor them via JSON return results.

Look into jQuery.ajax for some ideas on how to do this.

score 0 · Answer 3 · edited Mar 02 '21 at 19:22

To achieve the above mentioned functionality in a simple way, I would prefer calling a BackgroundWorker for each of the tasks. You can keep track of the progress plus you get a notification upon task completion. Another reason to choose this is that the mentioned tasks look like a back-end job and not tightly coupled with the UI. Here's a MSDN link and this is the link for a cool tutorial.

C# How to process several web requests at once

3 Answers3