22

Awhile ago I implemented some code to consume a REST Api using the HttpClient class.

using (var client = new HttpClient() { BaseAddress = new Uri(@"https://thirdparty.com") })
{
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(...);

    var uri = new Uri(@"rest/api/foo", UriKind.Relative);
    var content = new StringContent(json.ToString());

    using (var response = await client.PostAsync(uri, content))
    {
        // etc ...
    }
}

This code seemed to work perfectly fine against both the test and production environments (each of which access a test/production uri). Recently, we started to get an HttpRequestException in the production environment only: System.Net.Http.HttpRequestException: Error while copying content to a stream.

This seemed a bit strange, so I used Postman to send the same message and it worked just fine. I wasn't sure why our code was failing and Postman was working. I changed a parameter in the json data (the state from "NY" to "NV") and our .NET code worked fine -- of course we can't just send the wrong json data, so this isn't a solution; this was more of an observation that the exact same code worked fine with different content.

What's interesting is that there are two code changes we could make that will resolve this. Firstly, Postman is able to generate working C# code using the RestSharp package. Alternatively, I found an answer from another question pointing to using HttpVersion 1.0:

using (var request = new HttpRequestMessage(HttpMethod.Post, uri))
{
    request.Version = HttpVersion.Version10;
    request.Content = new StringContent(json.ToString());

    using (var response = await client.SendAsync(request))
    {
        // etc ...
    }
}

The confusing part is that Postman uses the HTTP/1.1 version. So, in summary:

  • If we change the json data (US State from "NY" to "NV"), the code works.
  • The same exact json and code works against the test uri.
  • Changing the code to use the RestSharp package works.
  • Changing the code to use HTTP/1.0 instead of HTTP/1.1 works.

Why on earth is Postman able to work using HTTP/1.1 but HttpClient fails? Is this a problem with our client (the code works for other US States)? Is this a bug in the .NET Framework? Is there something wrong with the implementation/hosting of the REST Api from the third party?

Postman headers:

POST rest/api/foo HTTP/1.1
Host: thirdparty.com
Content-Type: application/json
Authorization: Basic SOME_ENCRYPTED_USER_PASS
Cache-Control: no-cache
Postman-Token: 2fa5b5a0-b5d3-bd4c-40f0-d2b55b60316b

Sample Json:

{
    "stateCode": "NY",
    "packageID": "58330",
    "name": "58330-PRI-1",
    "documents": [
        {
            "type": "SPECIAL",
            "name": "Sample Document",
            "documentID": "3569976"
        }
    ],
    "descriptions": [
        {
            "city": "New York",
            "state": "NY"
        }
    ]
}

Stacktrace:

AggregateException: One or more errors occured.
HttpRequestException: Error while copying content to a stream.
IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
SocketException: An existing connection was forcibly closed by the remote host.
Community
  • 1
  • 1
myermian
  • 31,823
  • 24
  • 123
  • 215
  • Is the body of postman exactly the same as the body sent using client.PostAsync()? I believe the difference between versions is the keep-alive header so it could be Postman is handling timeouts differently to .NET. – Nick Spicer Jul 27 '16 at 14:24
  • @NickSpicer: Correct, the body is the same json data. And I believe it is the fact that it is HTTP/1.1 vs HTTP/1.0 that determines the connection keep-alive vs close. I've updated my answer to add the postman header data. – myermian Jul 27 '16 at 14:33
  • Have you tracked the details of the HttpRequestException ? – Rom Eh Jul 27 '16 at 14:47
  • I attempted to use Fiddler, but I can't seem to get it to work even when choosing to capture/decrypt HTTPS traffic. – myermian Jul 27 '16 at 15:04
  • Are you immediately using the response object after? Try using if (response.IsSuccessStatusCode) { } then logging the exception. – Nick Spicer Jul 27 '16 at 16:55
  • @NickSpicer: Yes, I do check the status code before anything else, but the exception happens on the line that is posting the async call, not on subsequent lines. – myermian Jul 27 '16 at 18:11
  • 1: When the exception occurs copy it to the clipboard and add it to the question. 2: Watch the traffic using Fiddler and see if it gives a response to the request with valid JSON in it. The difference could be that your app is trying to read the stream and convert it to C# objects whereas Postman won't be – Peter Morris Aug 01 '16 at 14:39
  • Have you tried setting the ContentType of the request explicitly: `request.Headers.ContentType = new MediaTypeHeaderValue("text/json");` – Tom Wuyts Aug 01 '16 at 14:41
  • @TomWuyts: No need to set content type if I use the solution with `HttpRequestMessage`, the problem is that if I don't use that (see first code sample -- there is no `request` object) and that is the code sample that does not work. – myermian Aug 01 '16 at 14:52
  • @PeterMorris: I tried Fiddler and I followed instructions on how to get https traffic, but it did not work. I did not want to spend too much time just to try to get Fiddler working so that I can debug this problem. Unfortunately I'm stuck without the help of Fiddler. – myermian Aug 01 '16 at 14:55
  • @m-y i think you better just use wireshark and do the same two request to any dummy http url that you have... watch the requests and compare the requests, its will tell you what exactly is going on... maybe post the two full requests here – Sufyan Jabr Aug 03 '16 at 04:06
  • 1
    are you using async/await all the way through? Strange things are known to happen when you mix async/await calls with their normal synchronised counterparts. – Duy Aug 03 '16 at 06:20
  • 1
    The fact it works when you change the data seems an evidence there's a problem with the server code. I don't think it's related to HTTP version (1.0 vs 1.1), or you have a very weird server broken an HTTP level. – Simon Mourier Aug 03 '16 at 07:04
  • @Duy: Yes, I use async/await the entire way through, and in fact I pulled out the code into LinqPad and isolated it down to militaristic code to isolate the problem away from anything else. – myermian Aug 03 '16 at 20:48
  • 1
    Please add the full stacktrace of the exception to your question. There might me helpful inner exceptions. – Good Night Nerd Pride Aug 04 '16 at 15:47
  • can you show us a sample json that you're sending? – Nikola.Lukovic Aug 08 '16 at 10:28
  • I second @Abbondanza 's request. It seems that not only you're withholding the stack trace, you're also withholding inner exceptions, which from your linked question: http://stackoverflow.com/questions/33233780/system-net-http-httprequestexception-error-while-copying-content-to-a-stream is `System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host`. means that the server is closing the connection at the TCP level. It's an important piece of information. – argaz Aug 08 '16 at 17:41
  • @Abbondanza: Sorry, didn't see your comment in the reply off all the comments. Updated the answer with the stacktrace. – myermian Aug 08 '16 at 18:05

2 Answers2

1

Due to the fact that you can change the data slightly and have it succeed, I would say your issues has nothing to do with any of your code. What if their server has cached a bad value and will continue sending you that value until their cache clears?

Try implicitly telling the server not to use cached values...

using (var client = new HttpClient() { BaseAddress = new Uri(@"https://thirdparty.com") })
{
    client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(...);

    var uri = new Uri(@"rest/api/foo", UriKind.Relative);
    var content = new StringContent(json.ToString());

    client.DefaultRequestHeaders.CacheControl = CacheControlHeaderValue.Parse("no-cache");

    using (var response = await client.PostAsync(uri, content))
    {
        // etc ...
    }
}

and see if you get a valid response stream.

Larry Dukek
  • 2,179
  • 15
  • 16
  • Sadly I get the same exception. The `"no-cache"` option did not change anything. – myermian Aug 03 '16 at 20:47
  • are you using response.Content.ReadAsStringAsync() or ReadAsStreamAsync/ReadAsByteArrayAsync ? – Larry Dukek Aug 03 '16 at 21:29
  • I'm not sure how that matters since the exception is thrown on the line executing `client.PostAsync(uri, content)`? Everything after that doesn't matter. – myermian Aug 04 '16 at 03:36
  • Just read something about the BaseAddress requiring a trailing forward slash at the end. It would surprise me if that was your issue as the code works with different data. Just mentioning it to cross that off the list. – Larry Dukek Aug 04 '16 at 13:50
  • One thing I would check is the json.ToString() value with the "NY" vs "NV" and see if there is some strange formatting issue with this specific set of data. Also I would try setting the content type: content.Headers.ContentType = new MediaTypeHeaderValue("application/json"); – Larry Dukek Aug 04 '16 at 13:56
  • Another thing you might try is pulling in (through nuget) the latest Microsoft.AspNet.WebApi.Client and use client.PostAsJsonAsync(uri,obj) and let it serialize your object into the JSON string and set the headers accordingly. – Larry Dukek Aug 04 '16 at 14:10
  • Or set the type in the StringContent constructor ... new StringContent(json.ToString(), System.Text.Encoding.UTF8, "application/json") – Larry Dukek Aug 04 '16 at 14:26
  • @TwistedStem, I agree with you.I was about to comment, and find yours :). .for using httpClient - http://stackoverflow.com/a/23095628/3142139 (answer by Fabiano) – M.Hassan Aug 06 '16 at 21:02
  • @TwistedStem: Sadly, all of these suggestions still don't solve the actual problem. The problem still occurs, and this doesn't really give me any clear indication if the problem is on my end (a bug in the Microsoft Code perhaps) or if the server is the problem. – myermian Aug 08 '16 at 16:21
0

Have you got Antivirus/Firewall on the machine running the HttpClient? I the past I have had problems with AVG, Mcafee, Norton and others silently blocking requests. Its quite tricky to find out exactly where, but there is may be a tab where ports are monitored, unticking / disabling this might help identify the problem. IF thats the case, the proper solution is to get your "thirdparty.com" on a white list of the appropriate vendor/s.

It may be worth looking at your response headers from your server as well? Could you add them to your question?Only because, in the past I have found Content-Security-Policy headers preventing some of my requests from completing?

Dai Bok
  • 3,451
  • 2
  • 53
  • 70