Throttled Http Client getting timeout after a large number of requests

Question

I am polling an API for the status of a large number of jobs (around 18k). I'm using a single instance of HttpClient, created like this:

var socketHandler = new SocketsHttpHandler
{
    MaxConnectionsPerServer = 4
};
_client = new HttpClient(socketHandler)

The method that calls the API using this HttpClient instance is passed as a Func to a rate limiting/throttling singleton, which uses a semaphore to only allow a given number of concurrent API calls:

public async Task<Tout> ExecuteWithLimitingAsync<Tout>(Func<Task<Tout>> func)
{
    _semaphore.WaitOne();
    try
    {
        return await func.Invoke();
    }
    finally
    {
        _semaphore.Release();
    }
}

The semaphore has a max and start value of 4. The problem is, after a large number of requests (above 100k for MaxConnectionsPerServer=4) I start getting System.TimeoutExceptions. First only some requests result in the exception, some result in a success, but after some time all of the requests end with an exception:

---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

The exceptions appear faster (after less requests) with higher values of allowed concurrent calls. I have tried manipulating the PooledConnectionLifetime/PooledConnectionIdleTimeout parameters, but it didn't help. There was also previously a Task.Delay() for around a second in the class responsible for throttling, but I have removed it when I reduced the number of concurrent calls.

From what I've read the HttpClient has an internal request queue, and requests waiting in this queue can time out if there's too many of them and they have to wait longer than their timeout value. However, here I'm limiting the numer of requests in queue to the same value as MaxConnectionsPerServer using the semaphore, so why would the error occur? Is it possible that HttpClient can't process 4 requests concurrently?

Might just be the server cannot cope with so many requests. Also the semaphore is not going to work properly with `async` because of thread identity issues, you should use `await SemaphoreSlim.WaitAsync` instead — Charlieface, Jul 27 '23 at 21:23
See this link : https://stackoverflow.com/questions/16194054/is-async-httpclient-from-net-4-5-a-bad-choice-for-intensive-load-applications — SoftwareDveloper, Jul 27 '23 at 22:13
Using SemaphoreSlim instead of Semaphore did the trick, thanks — gbrzeczek, Jul 28 '23 at 12:35

Throttled Http Client getting timeout after a large number of requests

0 Answers0