2

I am using .net HTTPClient across multiple multithreaded consumers making GetAsync web requests to a local service at 127.0.0.1 once per second.

The web requests complete 99.9% of the time but occasionally a few requests (over a 3-4 hour period) will get stuck in the GetAsyc and will not complete or timeout. Requests to the same service url/port in the same time period will work fine and new requests will complete fine.

The GetAsync is being fired off in a fire and forget mode where a callback is called on completion to handle the resulting parsed data (as it is integrated with some older code that doesn't use async.)

public void Execute(Action<IAsyncCommand> onCompletion)
{
    this.onAsyncCompletion = onCompletion;
    try
    {
       // do not await as this is fire and forget
       this.HandlRequestAysnc(this.Target, new StringContent(this.CommandPayload));
        return;
    }
    catch(Exception e)
    {
      //log exception
    }
 }
private async Task HandlRequestAysnc(Uri uri, StringContent stringContent)
{
    try
    {
        ConfiguredTaskAwaitable<HttpResponseMessage> request = stringContent != null ? webClient.PostAsync(uri, stringContent).ConfigureAwait(false) : webClient.GetAsync(uri).ConfigureAwait(false);
        //this will never return or timeout 1 in 10000 times
        using (HttpResponseMessage response = await request) 
        {
            if (response.IsSuccessStatusCode)
            {
                using (HttpContent content = response.Content)
                {
                    string result = await content.ReadAsStringAsync(); 
                    //handle result
                }
            }
            else
            {
             //handle failure
            }
        }
    }
    catch (Exception ex)
    {
         //log exception
    }
    if (this.onAsyncCompletion != null)
    {
        this.onAsyncCompletion(this);
    }
}
Ahmed Ashour
  • 5,179
  • 10
  • 35
  • 56
Grant G
  • 21
  • 4
  • Everything is alright here. Maybe the remote server is simply not responding? Set a timeout and test that the timeout works. Confirm to me that you tested the timeout. – usr May 22 '15 at 20:00
  • I have tested the timeout and it works correctly during testing. I can block my service with a breakpoint and will receive a timeout in my .net code. The HTTPClient is set up as follows: private HttpClient webClient = new HttpClient(new HttpClientHandler() { AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip }) { MaxResponseContentBufferSize = 10000000, Timeout = TimeSpan.FromSeconds(150) }; – Grant G May 22 '15 at 22:28
  • Are you using Result or Wait anywhere in the app? – usr May 22 '15 at 22:35
  • No--the task is never waited and we never access Result. We have tried to look for the deadlock condition but it doesn't seem to exist. This also only seems to happen every 10000 or so requets... – Grant G May 22 '15 at 22:43
  • We ended up adding some code to inspect the running tasks (which now take a cancel token) and cancel them if they have been stuck for too long. This seems to work as the will return from GetAsync... Though I would still like to get to the bottom of this one. – Grant G May 22 '15 at 23:31

1 Answers1

0

One of the issues with GetAync is that the TCP stack is now under control as soon as a session is started. A recent lab experiment proved that spinning up 10,000 get requests (for investigating why we were having memory issues in an prod environment), took over 5 minutes (After the application was ended) for the TCP stack to clean up everything.

If you find that the socket status has Fin-Wait 1 or 2, Time-Wait or other half-baked sessions it's simply a symptom of a larger issue which is one of the two systems (or both) cannot handle the traffic at that rate. Once this starts happening things grow out of control quickly as both sides are fighting to maintain the sessions but are losing the resources to do it fast enough.

The solution to this type of issue is to find another way to increase the throughput.

JWP
  • 6,672
  • 3
  • 50
  • 74