Static HttpClient but still too many open sockets

Question

I'm using static Lazy<HttpClient>, which should ensure that sockets are reused. (You're using HttpClient wrong and it could be destabilizing your software)

However, my application freezes after a few hundred requests. I've added trace logging and could reproduce the issue in an MVC action that does nothing* but a call to HttpClient.

To be explicit, it reads a web.config value, validates some ViewModel values, and parses the HttpClient response, but that never comes in.

netstat shows that there are a lot of open sockets. Even now, minutes later, after I have stopped the requests and have spent some time writing this question, the number of sockets does not seem to have decreased.

C:\Users\me>netstat

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    127.0.0.1:443          localhost:53730            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53731            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53732            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53733            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53734            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53736            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53737            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53738            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53739            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53740            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53741            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53743            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53751            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53752            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53753            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53754            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53755            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53760            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53765            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53766            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53767            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53768            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53769            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53773            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53776            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53777            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53778            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53779            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53780            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53784            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53787            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53788            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53789            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53790            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53791            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53795            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53798            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53799            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53800            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53801            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53802            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53806            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53808            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53809            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53810            TIME_WAIT
  TCP    127.0.0.1:443          localhost:53811            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53812            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53813            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53814            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53815            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53816            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53817            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53818            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53819            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53820            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53821            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53822            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53823            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53824            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53825            CLOSE_WAIT
  TCP    127.0.0.1:443          localhost:53851            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53852            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53853            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53854            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53855            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53856            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53857            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53858            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53859            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53860            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53861            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53862            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53863            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53864            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53865            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53866            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53867            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53868            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53869            ESTABLISHED
  TCP    127.0.0.1:443          localhost:53870            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53301            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53838            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53839            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53840            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53841            ESTABLISHED
  TCP    127.0.0.1:44117        localhost:53842            ESTABLISHED
  TCP    127.0.0.1:49676        localhost:49677            ESTABLISHED
  TCP    127.0.0.1:49677        localhost:49676            ESTABLISHED
  TCP    127.0.0.1:50498        localhost:50501            ESTABLISHED
  TCP    127.0.0.1:50501        localhost:50498            ESTABLISHED
  TCP    127.0.0.1:53301        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53697        localhost:53698            ESTABLISHED
  TCP    127.0.0.1:53698        localhost:53697            ESTABLISHED
  TCP    127.0.0.1:53699        localhost:53700            ESTABLISHED
  TCP    127.0.0.1:53700        localhost:53699            ESTABLISHED
  TCP    127.0.0.1:53701        localhost:53702            ESTABLISHED
  TCP    127.0.0.1:53702        localhost:53701            ESTABLISHED
  TCP    127.0.0.1:53703        localhost:53704            ESTABLISHED
  TCP    127.0.0.1:53704        localhost:53703            ESTABLISHED
  TCP    127.0.0.1:53705        localhost:53706            ESTABLISHED
  TCP    127.0.0.1:53706        localhost:53705            ESTABLISHED
  TCP    127.0.0.1:53707        localhost:53708            ESTABLISHED
  TCP    127.0.0.1:53708        localhost:53707            ESTABLISHED
  TCP    127.0.0.1:53709        localhost:53710            ESTABLISHED
  TCP    127.0.0.1:53710        localhost:53709            ESTABLISHED
  TCP    127.0.0.1:53711        localhost:53712            ESTABLISHED
  TCP    127.0.0.1:53712        localhost:53711            ESTABLISHED
  TCP    127.0.0.1:53713        localhost:53714            ESTABLISHED
  TCP    127.0.0.1:53714        localhost:53713            ESTABLISHED
  TCP    127.0.0.1:53715        localhost:53716            ESTABLISHED
  TCP    127.0.0.1:53716        localhost:53715            ESTABLISHED
  TCP    127.0.0.1:53717        localhost:53718            ESTABLISHED
  TCP    127.0.0.1:53718        localhost:53717            ESTABLISHED
  TCP    127.0.0.1:53719        localhost:53720            ESTABLISHED
  TCP    127.0.0.1:53720        localhost:53719            ESTABLISHED
  TCP    127.0.0.1:53721        localhost:53722            ESTABLISHED
  TCP    127.0.0.1:53722        localhost:53721            ESTABLISHED
  TCP    127.0.0.1:53723        localhost:53724            ESTABLISHED
  TCP    127.0.0.1:53724        localhost:53723            ESTABLISHED
  TCP    127.0.0.1:53725        localhost:53726            ESTABLISHED
  TCP    127.0.0.1:53726        localhost:53725            ESTABLISHED
  TCP    127.0.0.1:53727        localhost:53728            ESTABLISHED
  TCP    127.0.0.1:53728        localhost:53727            ESTABLISHED
  TCP    127.0.0.1:53730        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53735        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53742        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53744        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53745        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53746        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53747        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53748        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53749        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53750        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53751        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53758        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53761        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53762        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53763        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53765        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53770        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53771        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53772        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53773        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53775        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53781        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53782        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53783        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53784        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53786        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53792        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53793        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53794        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53795        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53797        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53803        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53804        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53805        localhost:https            TIME_WAIT
  TCP    127.0.0.1:53806        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53808        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53811        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53812        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53813        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53814        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53815        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53816        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53817        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53818        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53819        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53820        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53821        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53822        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53823        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53824        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53825        localhost:https            FIN_WAIT_2
  TCP    127.0.0.1:53838        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53839        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53840        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53841        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53842        localhost:44117            ESTABLISHED
  TCP    127.0.0.1:53851        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53852        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53853        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53854        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53855        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53856        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53857        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53858        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53859        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53860        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53861        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53862        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53863        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53864        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53865        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53866        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53867        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53868        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53869        localhost:https            ESTABLISHED
  TCP    127.0.0.1:53870        localhost:https            ESTABLISHED
snip

I have two occurrences of HttpClient, both with a separate BaseAddress. I tried replacing HttpClient with Easy.Common.RestClient, to no avail.

I've set

var servicePoint = ServicePointManager.FindServicePoint(new Uri(webApiAddress));
servicePoint.ConnectionLeaseTimeout = (int)TimeSpan.FromMinutes(1).TotalMilliseconds;

but that doesn't help either. I've set the timeout to 15 seconds. I've tried setting CloseConnection=true or setting header "Connection":"Close"

I don't know what to try anymore... Can someone provide guidance?

Update

Let's go into details. I have a static Lazy<HttpClient> in a class library, inside a class MyClientWrapper (example). My Asp.Net MVC 5 application includes this class library and instantiates MyClientWrapper.

My MVC controller calls _myClientWrapper.GetData(). Inside GetData, an HttpRequestMessage is created, sent to my API using HttpClient. The HttpResponseMessage is read. Both request and response are disposed and the data is returned from GetData.

I am using Gatling to fire 20 requests for 20 users to MyController/MyAction. At some point (sometimes earlier, sometimes later), no requests arrive at the API anymore. Putting a debug breakpoint in MyController shows that GetData() is hit. The await call is hit and that's it. netstat fills up.

I've tried to break down the problem.

Scenario 1: ping the API using PostMan (before anything has hung up on me). No issues.

Scenario 2: Delete all code from GetData() and replace by return somedummydata.

public async Task<Data> GetData()
{
    // No call to HttpClient, no issues
    return new Data();
}

Scenario 3: In GetData(), just ping the API using HttpClient, nothing else and still return some dummy data. The problem occurs After a few pings, nothing reaches the API anymore. Stepping through MyController shows that the GetData call is hit, but it doesn't come through. After everything has frozen up, I can't ping the API using PostMan either. Nothing comes through (trace logging on Application_BeginRequest in API does not appear, so the API isn't even reached..)

public async Task<Data> GetData()
{
    // Issue occurs as soon as HttpClient is used for GET or POST. In this example, GET.
    await _client.GetAsync("pseudocode/Ping");
    // In reality I properly HttpRequestMessage/HttpResponseMessage read and dispose, this is just pseudocode really... 
    return new Data();
}

Update

Changing HttpClient from static to PerThread (Castle Windsor IoC) makes no difference
Interestingly, whenever the hangup occurs, there are exactly three requests in IIS under WorkerProcesses. Every single time. Exactly three. Can this have anything at all to do with the maximum number of connections (default 2) that HttpClient has?
I've verified that my API's ping method has no logic inside. It does not.

Update

Perhaps related: HTTP connection idles out then times out when running more HttpClient.GetAsync() parallel tasks than ServicePointManager.DefaultConnectionLimit

So in other words, this usage is destabilizing your software? J/k, I'm sure it's fine. Did you run `netstat` before you started your test? A few things... you're connecting to a server from a client on the same machine, so all of your connections will be listed twice... any other connection from/to the same machine will be also be listed twice (like every time you open the page in your browser to make sure the server is still responding)... just because you close the connection on "your end" doesn't mean the server will. From the article you linked, you should test against their Azure server. — rfmodulator, Apr 10 '20 at 15:29
Thanks for your reply and for pointing out they're listed twice. I did run `netstat` before. I recycled all app pools, restarted my application, rebooted my laptop. Only 1 socket in use in `netstat`. Then, started my test and boom. I do not use Azure, but I may be able to test against a separate server later today. — Diana, Apr 10 '20 at 15:38
I don't think this `static Lazy` usage is going to behave as you expect in an MVC Application. The examples in the article, and linked pages, seem to all be Desktop (Console) Applications. — rfmodulator, Apr 10 '20 at 15:56
With my search history of a few weeks ago as a witness.. I was kinda worried about that. I admit I haven't dug into the behavior of `static` in MVC and AppPools. Do you have any suggestions or is it mostly a hunch? — Diana, Apr 10 '20 at 16:00
It's a hunch at best... based on my own limited (almost non-existent) understanding of IIS AppPools. I'm going to upvote, because I'd like to know as well, and add a few tags that may attract the correct people to answer this. — rfmodulator, Apr 10 '20 at 16:07
`TIME_WAIT` is not an open socket. It's a socket that has been closed but the OS has to keep reserved for a while to cleanly discard any packets that arrive intended for the now-closed connection. Ideas for fix: https://stackoverflow.com/a/1931378/263693 — Stephen Cleary, Apr 10 '20 at 21:56
Please scroll down, it's a mix of everything, not just `TIME_WAIT`. But I will look into the link you gave of course — Diana, Apr 11 '20 at 08:34

Static HttpClient but still too many open sockets

0 Answers0