Asynch and Parallel HttpClient Request to 62,000+ machines : Best Practice Request

Question

I have this scenario, I am working on where a notification message needs to be sent to all workstations in my company. We have a client service which handles Desktop Notifications and it is a very simple WebApi.

Question/ Requirement How do I both Asynchronously and in Parallel send out request to all these machines from server/ASP.Net web app, and capture its response to custom log file?

Many machines may be switched off at the time of notification is sent out or dns might not be able to resolve as the machine could have been decommissioned.

The request/response cycle to the code which is currently being prototyped needs to a non-blocking one. So that the user does not have to wait for response from all these machines.

My Homework: I understand Asynchronous operations are better suited for IO bound operations, however the number of machines are so many that makes me feel parallel and async together, might suit this situation better.

I have set ServicePointManager.DefaultConnectionLimit = 10000

HttpClient with constructor with dispose option to false has been passed. See Below code - Factory Class.

HttpClient class in framework is designed for reuse however, from what I understand some properties not be changed like base address. Source : HttpClient - This instance has already started

I am using Request/Response specific ID (Correlation ID) to make sure all async and parallel operations right to the same folder for a given request.

TaskCanceledException catch block never gets hit !!

Timeout has been extended to 30 seconds which I am quite sure is more than enough.

Prototype Code:

public class HttpClientFactory  : IHttpClientFactory
    {
        public void CreateClient(string baseUrl, Action<HttpClient> methodToExecute)
        {
            using (var handler = new HttpClientHandler())
            {
                handler.AllowAutoRedirect = true;
                using (var client = new HttpClient(handler, false))
                {
                    client.BaseAddress = new Uri(baseUrl);
                    client.Timeout = TimeSpan.FromSeconds(30);
                    methodToExecute(client);
                }
            }
        }
   }

Where IHttpClientFactory is Transient (IoC)

workstationUrls.ForEach(baseUrl =>
            {
                _httpClientFactory.CreateClient(baseUrl, async (client) =>
                {
                    await client.PostAsync(resourceUrl, content).ContinueWith(t =>
                    {
                        try
                        {
                            var response = t.Result;

                            var workstationResponse = new WorkstationResponse
                            {
                                StatusCode = (int)response.StatusCode,
                                Response = response.Content.ReadAsStringAsync().Result
                            };

                            workstationResponse.IsSuccess
                                = workstationResponse.StatusCode >= 200 &&
                                  workstationResponse.StatusCode <= 299
                                    ? true
                                    : false;

                            var docContent = JsonConvert.SerializeObject(workstationResponse);
                            if (workstationResponse.IsSuccess)
                            {
                                // Write workstation log
                                File.WriteAllText(path
                                                  + "\\Bulk Notifications\\"
                                                  + userFolder + "\\Success\\"
                                                  + GetWorkstationNameFromUrl(baseUrl)
                                                  + GetUniqueTimeStampForFileNames()
                                                  + workstationResponse.IsSuccess + ".txt",
                                    docContent);
                            }
                            else
                            {
                                // Write workstation log
                                File.WriteAllText(path
                                                  + "\\Bulk Notifications\\"
                                                  + userFolder + "\\Fail\\"
                                                  + GetWorkstationNameFromUrl(baseUrl)
                                                  + GetUniqueTimeStampForFileNames()
                                                  + workstationResponse.IsSuccess + ".txt",
                                    docContent);
                            }

                        }
                        catch (TaskCanceledException exe)
                        {

                        }
                        catch (Exception ex)
                        {

                            var workstationResponse = new WorkstationResponse
                            {
                                Exception = ex,
                                IsSuccess = false
                            };
                            var docContent = JsonConvert.SerializeObject(workstationResponse);
                            // Write workstation log
                            File.WriteAllText(path
                                              + "\\Bulk Notifications\\"
                                              + userFolder + "\\Fail\\"
                                              + GetWorkstationNameFromUrl(baseUrl) + " "
                                              + GetUniqueTimeStampForFileNames() + " "
                                              + workstationResponse.IsSuccess + ".txt", docContent);
                        }
                    });
                });
            }); 

Problem With Prototype Code: I get these errors in Exception messages and StackTrace does not seem to be helpful

• "A task was canceled"
• "Cannot access a disposed object.Object name: 'System.Net.Http.StringContent'"

Answer 1

First, async and parallel are not mutually exclusive; the difference is in the constructs you use and in the fact that unlike "traditional" parallelism, you're not consuming/blocking a thread per async operation.

The most basic construct for executing async operations concurrently and waiting (asynchronously) for all of them to complete is Task.WhenAll. I'd start by defining an async method for sending a message to a single workstation:

async Task SendMessageToWorkstationAsync(string url)

(The implementation can be derived pretty easily from your code.)

Then call it like this:

await Task.WhenAll(workstationUrls.Select(SendMessageToWorkstationAsync));

Second, regarding HttpClient, if you're not reusing a single instance because you're setting BaseAddress, the solution is simple: don't do that. :) It's not required. Skip the factory class and just create a single shared HttpClient instance instead, and provide a full URI with each request.

Finally, depending on a variety of factors, you might still find that 62,000+ concurrent requests is more than the system can handle. If that is the case, you'll want to throttle your parallelism. I've found the best way to do that is with TPL Dataflow. See this question for details on how to do that.