Run multiple WebRequest in parallel in asp.net core

Question

I have the following situation. I want to execute 6 URL request (each request is with different auth_token and returns different data ). Here is the function that is making requests for different auth_token.

 private static async Task<List<Positions>> GeneratePositions(string auth_token)
        {
            var request = WebRequest.Create(www.some_url.com);

            request.Headers.Add("authorization", "Bearer " + auth_token);
            request.Method = "GET";

            WebResponse response = null;

            try
            {
                response = await request.GetResponseAsync();
            }
            catch (Exception ex)
            {
                // TODO log
                throw ex;
            }

            var dataStreamRes = response.GetResponseStream();
            var reader = new StreamReader(dataStreamRes);
            var buffResponse = await reader.ReadToEndAsync();

            var positions = JsonSerializer.Deserialize<List<Positions>>(buffResponse);

            reader.Close();
            dataStreamRes.Close();
            response.Close();

            return positions;
        }

This function is ran using await operator. But the problem is that every requests waits for the previous one to start. Do you have any idea how I can run multiple requests for multiple auth_tokens at the same time, without waiting the previous one to finish, and of course collect the data returned.

hmm, maybe problem is in the place where you call your function ? — Bartosz Olchowik, Mar 30 '22 at 07:04
Might be helpful: [ForEachAsync with Result](https://stackoverflow.com/questions/30907650/foreachasync-with-result) — Theodor Zoulias, Mar 30 '22 at 07:25

Jeroen van Langen · Answer 1 · 2022-03-30T07:19:15.803

The position to make it parallel is not inside the method you've shown, but where you'd call it.

Instead of calling this method like:

// for example sequential:
foreach(var siteInfo in siteInfoList)
{
    // await per call.
    var positions = await GeneratePositions(siteInfo.url, siteInfo.auth_token);

    foreach(var position in positions)
    {
        Console.WriteLine(position.XXX);
    }
}

You can use the Task.WhenAll() method. This might run the tasks parallel.

// create a list of tasks
var tasks = new List<Task<List<Positions>>>();

foreach(var siteInfo in siteInfoList)
{
    // add the task to a list of tasks
    tasks.Add(GeneratePositions(siteInfo.url, siteInfo.auth_token));
}

// await them all.
await Task.WhenAll(tasks);

// read back the results.
foreach(var task in tasks)
{
    foreach(var position in task.Result)
    {
        Console.WriteLine(position.XXX);
    }
}

A big difference between these two options is exception handling.

Okay, but how can I accumulate the returned List with this approach ? — pewocis495, Mar 30 '22 at 09:36

Run multiple WebRequest in parallel in asp.net core

1 Answers1