HTTP client Get request within a threading parallel pool - random failures

Question

I have an API which gives me status of a few sensors (located on the Azure functions)

www.myapp.com/api/sensors/{sensor_id}

My client application needs to check each sensor every one second! so in this case I have 10 sensors and the client need to send 10 http get request every second!

I decided to do it using Parallel and create a pool out of each instance of my method which runs an executes the Get request!

This is the main method which orchestrate and add each sensor to the pool and invoke them all


        public override Task Run(List<int> sensors)
        {
            try
            {
                List<Action> actions = new List<Action>();
                foreach (var prd in sensors)
                {
                    actions.Add(async () => await Process(prd));
                }
                ParallelOptions parallelOptions = new ParallelOptions
                {
                    MaxDegreeOfParallelism = 20 
                };
                Parallel.Invoke(parallelOptions, actions.ToArray());
            }
            catch (Exception ex)
            {
                Run(sensors);
            }
            return Task.CompletedTask;
        }

In the Process() method I get the sensor Id and add it to the API endpoint then run the Get request every 1 second.


        private async Task<string> Process(int sensorId)
        {
            while (true)
            {
                Thread.Sleep(1000);
                try
                {
                    SensorModel response = _httpClientService.GetAsync<SensorModel>(string.Format(apiUrl, sensorId)).Result;
                    if (response != null)
                    {
                        if (response.Status == "success")
                        {
                      
                        .
                        .
                        .

                    }
                        _logger.LogError($"API connection unsuccesful...");
                        return null;
                    }
                    _logger.LogError($"API connection failed...");
                    return null;
                }
                catch (Exception ex)
                {
                    _logger.LogError($"Exception in retrieving sensor data {StringExtensions.GetCurrentMethod()} {ex}");
                    return null;
                }
            }

        }

and my GetAsync in my HttpClient factory

        public async Task<T> GetAsync<T>(string uri)
        {
            string responseData = null;
            try
            {
                var response = await _client.GetAsync(uri);
                responseData = await response.Content.ReadAsStringAsync();
                if (!string.IsNullOrEmpty(responseData) && response.StatusCode == HttpStatusCode.OK)
                    return JsonConvert.DeserializeObject<T>(responseData);
                return default;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, responseData);
                return default;
            }
        }

Now the above code works fine most of the time, but soemtimes it just starts failing the connections and returning the exception below, the only way I can fix it is to restart the application. Not sure it maybe getting confused with a lot of requests and the fact that they are being processed simultaneously

System.ArgumentNullException: Value cannot be null. (Parameter 'obj')
   at System.OrdinalIgnoreCaseComparer.GetHashCode(String obj)
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Net.Http.Headers.HttpHeaders.AddHeaders(HttpHeaders sourceHeaders)
   at System.Net.Http.Headers.HttpRequestHeaders.AddHeaders(HttpHeaders sourceHeaders)
   at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.GetAsync(String requestUri)
   at SensorMonitor.Services.Implementations.HttpClientService.GetAsync[T](String uri)

I know it is within my GetAsync method but can't figure out why it is happening after a few minutes of running and not just from the start!

Also note I had to make my GetAsync method not to be async by calling .Result as if I do await then the app crashes before even it starts the first request! It seems it doesn't like being called in a Parallel Pool.

I'd appreciate your help!

Updated with more details:

HttpClientService.cs


    public class HttpClientService : IHttpClientService
    {
        private readonly HttpClient _client;
        private readonly ILogger<HttpClientService> _logger;
        private static readonly JsonSerializerSettings JsonSerializerSettings = new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore };


        public HttpClientService(HttpClient client, ILogger<HttpClientService> logger)
        {
            _client = client;
            _logger = logger;
        }
.
.
.

And this is how I registered my HttpClient

            services.AddHttpClient<IHttpClientService, HttpClientService>();

How inject HttpClient into your HttpClientService? and how add your service to DI? please update your question — sa-es-ir, Mar 10 '21 at 19:42
Instead of using ``Thread.Sleep(1000);`` use this ``await Task.Delay(1000);`` and test it — sa-es-ir, Mar 10 '21 at 20:03
Didn't work, it went through one loop then existed, I think that's because after the delay it returns a Task and when the Run method gets a Task it assumes the application is done with the execution! — user65248, Mar 10 '21 at 20:07
No. when use Thread.Sleep, it block whole thread and it's not good. have you try to use List and ``await Task.WhenAll`` instead of Parallel? — sa-es-ir, Mar 10 '21 at 20:12
Maybe this helped https://stackoverflow.com/questions/24306620/parallel-invoke-does-not-wait-for-async-methods-to-complete — sa-es-ir, Mar 10 '21 at 20:17
But that's my expected behaviour, I need the thread to be paused and blocked for one second on that single particular thread! Parallel generated 10 threads for me! Each of them are independent and I need each of them to be blocked and paused for one second before the Http request! — user65248, Mar 10 '21 at 20:19
you want sleep each created thread not ``Main`` thread that your app run it — sa-es-ir, Mar 10 '21 at 20:21
True, that's exactly what is happening here, the main thread is the one that Run is executed on and parallel's job is to run each Process() method on a different thread! — user65248, Mar 10 '21 at 20:30

score 1 · Accepted Answer · answered Mar 11 '21 at 14:41

Here's a few general guidelines:

Avoid async void. Your code is currently adding async lambdas as Action delegates, which end up being async void.
Go async all the way. I.e., don't block on async code. In particular, don't use .Result.

Finally, Parallel doesn't mix with async, because Parallel is for CPU-bound multi-threaded parallelism, not for I/O-bound unthreaded asynchrony. The solution you need is asynchronous concurrency, i.e., Task.WhenAll.

public override async Task Run(List<int> sensors)
{
  var tasks = sensors
      .Select(prd => Process(prd))
      .ToList();
  await Task.WhenAll(tasks);
}

private async Task<string> Process(int sensorId)
{
  while (true)
  {
    await Task.Delay(1000);
    try
    {
      SensorModel response = await _httpClientService.GetAsync<SensorModel>(string.Format(apiUrl, sensorId));
      if (response != null)
      {
        if (response.Status == "success")
        {
          ...
        }
        _logger.LogError($"API connection unsuccesful...");
        return null;
      }
      _logger.LogError($"API connection failed...");
      return null;
    }
    catch (Exception ex)
    {
      _logger.LogError($"Exception in retrieving sensor data {StringExtensions.GetCurrentMethod()} {ex}");
      return null;
    }
  }
}

Thanks very much, I didn't realise about the async void! It's now working as expected! — user65248, Mar 11 '21 at 19:06

HTTP client Get request within a threading parallel pool - random failures

1 Answers1