How to send http requests in ForEach loop with async / await

Question

The following code is a linqpad program.

• The payloadList contains json-objects like {"id": 1, "foo": "one" }.
• Each object of payloadList should be sent to a server with httpClient.SendAsync()
• The response for each request should be stored in responseList

The code below does partly work. But i do not understand why some parts are not working. I assume that the responses are not completed when responseList.Add(foo) is executed.

This request shall be send for each json-object {"id": 1, "foo": "one" }

public static async Task<string> DoRequest(HttpClient client, string payload)
{   
    var request = new HttpRequestMessage(HttpMethod.Post, 
                           "http://httpbin.org/anything");
    request.Content = new StringContent(payload
                           , Encoding.UTF8, "application/json");        
    var response = await client.SendAsync(request); 
    string responseContent = await response.Content.ReadAsStringAsync(); 
    return responseContent;
}

The DoRequest()-method wraps the request and can be used inside main like this

static async Task Main()
{
    var responseList = new List<string>();  
    var payloadList = new List<string>{"{ 'id': 1, 'bar': 'One'}",
                                       "{ 'id': 2, 'bar': 'Two'}",
                                       "{ 'id': 3, 'bar': 'Three'}"};
        
    var client = new HttpClient();
    
    payloadList.ForEach(async (payload) => {
        var responseFoo = await DoRequest(client, payload);
        responseFoo.Dump("response"); // contains responseFoo
        responseList.Add(responseFoo);  // adding responseFoo to list fails
    });                     
    responseList.Dump();    // is empty
}

The responseList is empty.

• Expected responseList.Dump() contains all responses responseFoo.
• Actual responseList is empty.

Questions

• How can each response for await client.SendAsync(request) be added to a responseList?
• Why is responseList empty despite that foo.Dump() works?
• How to confirm or check if every client.SendAsync is finished?
• Would you write the code above different - why?

Answer 1

List.ForEach is not Task-aware and will execute everything in parallel without waiting for the results (i.e. it will create tasks for all items in payloadList and will continue to the next statement responseList.Dump(); without awaiting them).

1. In newer versions of .NET you can use Parallel.ForEachAsync(for example as in this answer) combined with use of appropriate collection from System.Collections.Concurrent, for example ConcurrentBag<T>. List is not thread-safe and modifying it concurrently can lead to a lot of problems.

   Code can look like the following:

   var responseList = new ConcurrentBag<string>();
   
   await Parallel.ForEachAsync(payloadList, async (payload, ct) =>
   {
       var foo = await DoRequest(client, payload, ct);
       responseList.Add(foo);     
   });
   
   static async Task<string> DoRequest(HttpClient client, string payload, CancellationToken ct)
   {   
       var request = new HttpRequestMessage(HttpMethod.Post, "http://httpbin.org/anything");
       request.Content = new StringContent(payload, Encoding.UTF8, "application/json");        
       var response = await client.SendAsync(request, ct); 
       string responseContent = await response.Content.ReadAsStringAsync(ct); 
       return responseContent;
   }
   

2. If you are fine with all requests running in parallel - just create a enumerable of tasks and use Task.WhenAll<T>():

   var tsks = payloadList
       .Select(payload => DoRequest(client, payload));
   
   string[] result = await Task.WhenAll(tsks);
   

3. If you want to execute requests one after another - just switch to ordinary foreach:

   var responseList = new List<string>(); // no need for concurrent collection in this case 
   
   foreach (var payload in payloadList)
   {
       var foo = await DoRequest(client, payload);
       responseList.Add(foo);  
   }  
   

If you want to dive deeper here some links:

• Asynchronous programming with async and await documentation
• Asynchronous Programming in .NET - Introduction, Misconceptions, and Problems
• Threading in C# by Joseph Albahari (a bit dated but still amazing)