I have a strange issue I'm trying to triage having to do with the new HttpClient on .NET Core 2.1. From this article here (https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/) I know that the HttpClient has been completely re-written to use a different low level library for handling HTTP requests. I'm wondering if anyone has seen any issues with the new implementation.
What I'm seeing is a strange case where my application (.NET Core 2.1) which sends a POST request to some API periodically (every 10 seconds) a few of times every 15 min it will throw an exception with the error: An error occurred while sending the request.; The server returned an invalid or unrecognized response.
No other details are available, it's just an exception when I make a call like this:
using (var res = await _httpClient.PostAsync(uriBuilder.Uri, new StringContent(serializedRequestBody, Encoding.UTF8, "application/json")))
{
//Do something here
}
The exception caught is a System.Net.Http.HttpRequestException and it has some inner exception with the above error message.
So as I mentioned this does NOT happen all time, it happens seemingly at random, or at least I can not discern any particular pattern. All I can say is these POST requests are made once every 10 seconds 24/7 and anywhere between 5% and 10% of the POST requests fail with the above exception.
So used tcdump and piped it into wireshark to examine the requests to see what's actually happening when the requests fail and what i see is the following:
On a good POST I see: my app sends the request to server, server sends response back, my app sends ACK to server and server responds with FIN,ACK. Done. Good Stuff.
On POST which gets the above exception I see the following: my app sends the request to server, and almost immediately after (like a few milliseconds after) my application sends FIN, ACK to server.
This seems consistent with what I see in my application logs, which show that the request duration is 0 before the exception is thrown.
So what it looks like to me is, my application sends the request and then immediately after closes the connection for some reason. However, I don't understand why this happens. I tried comparing the raw HTTP requests (good POST vs bad POST) to see any differences and I can not see any difference.
One last thing to mention, is that I ONLY see this in applications running on .NET Core 2.1. When I run my application on .NET 2.0 I do not see this problem. Also when I use the same library (where the HTTP call is being made) in the .NET 4.5.1 application (I use multi-targeting to compile the library targeting .net standard and net451) I also do NOT see this problem. So it seems to affect only .NET Core 2.1
Any ideas of where I can go from here? Is there something else I should look for ? How would someone go about trying to triage this type of issue ?
[EDIT] I added a screenshot of the wireshark output which shows the last POST request the server never does not respond before the client sends FIN,ACK
[EDIT] @Svek Pointed out something in the comments about the sequence of ACKs. I think there maybe something here, because (in the screenshot) after the very last POST there is a FIN, ACK and it shows Ack=7187, so I look back I see the previous FIN,ACK had sequence=7186. Now, I'm by far not an expect in TCP or networking so I maybe saying something completely dumb, but does that mean that the last FIN,ACK (which comes from my host to server) is essentially my host FIN,ACK'ing the previous FIN,ACK (from server to my host) and essentially closing the connection.
So since the next POST is made to the same host:port, using the same connection and yet the connection is closed (via that last FIN,ACK) that's why I never get a response back?
