1

We are running a WebJob and Api on Azure App Services. Some of the WebJobs perform REST Calls to third party services, like ebay. All worked fine, until a few days ago, when the services started throwing this error randomly:

{\"ClassName\":\"System.Net.Http.HttpRequestException\",\"Message\":\"An error occurred while sending the request.\",\"Data\":{},\"InnerException\":{\"ClassName\":\"System.Net.WebException\",\"Message\":\"The underlying connection was closed: An unexpected error occurred on a receive.\",\"Data\":{},\"InnerException\":{\"ClassName\":\"System.IO.IOException\",\"Message\":\"Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\",\"Data\":{},\"InnerException\":{\"NativeErrorCode\":10060,\"ClassName\":\"System.Net.Sockets.SocketException\",\"Message\":\"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\",\"Data\":{},\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":\"   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)\\r\\n   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)\",\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":\"8\\nEndReceive\\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\\nSystem.Net.Sockets.Socket\\nInt32 EndReceive(System.IAsyncResult)\",\"HResult\":-2147467259,\"Source\":\"System\",\"WatsonBuckets\":null},\"HelpURL\":null,\"StackTraceString\":\"   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)\\r\\n   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)\\r\\n   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)\",\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":\"8\\nEndRead\\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\\nSystem.Net.Security._SslStream\\nInt32 EndRead(System.IAsyncResult)\",\"HResult\":-2146232800,\"Source\":\"System\",\"WatsonBuckets\":null}

The calls sometimes work, but are very slow and sometimes return the error. Running a local instance of the service results in no failure. Only in the production environment, we have these issues.

We use a singleton instance of the HttpClient to perform the calls.

public sealed class Client : HttpClient
    {
        private static volatile Client _instance = new Client();

        static Client()
        {
        }

        private Client() : base(new NativeMessageHandler())
        {
            // limit the connections in parallel to 100 by default
            ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;

            // if this setting does not work, follow these instructions on app.config
            // ServicePointManager.DefaultConnectionLimit needs to be set before the ServicePoint is created 
            ServicePointManager.DefaultConnectionLimit = 100;
        }

        public static Client Instance => _instance;
    }

We call the endpoint using the Client like this:

var client = Client.Instance;
var authenticationHeader = new AuthenticationHeaderValue("Bearer", token.AuthToken);
var url = "https://api.ebay.com/sell/account/v1/fulfillment_policy?marketplace_id=EBAY_DE";
var response = await client.GetMessageAsync(url, m => m.Headers.Authorization = authenticationHeader);

The GetMessageAsync Method is an Extension Method and just performs the action to set the header.

The problems started shortly after Microsoft announced this Security patch: https://learn.microsoft.com/answers/questions/6842/announcement-samesite-cookie-handling-and-net-fram.html

The Client is set to accept TLS 1.2 and 1.1.

Michael Staples
  • 537
  • 7
  • 13
  • Try keeping only tls11 and tls12 in the service point manager. – Soumen Mukherjee Feb 22 '20 at 13:39
  • @SoumenMukherjee Gave it a try with the same result. Local, no problem, on Azure, Time Out. – Michael Staples Feb 22 '20 at 13:48
  • OK I see that ebay API are also supporting TLS 1.3 , again a trial an error approach , can you also enable TLS13 . And i think when the connection is getting established from the local machine it is using TLS 1.3 – Soumen Mukherjee Feb 22 '20 at 14:03
  • https://stackoverflow.com/questions/55240173/how-to-handle-httpwebrequest-c-sharp-with-tls-1-3 TLS1.3 is not supported by the .NET Framework version we use. 4.8 is super new and updating the project seems a bit drastic at the moment. But, if it were a TLS1.3 compatibility issue, wouldn't I run into this issue on local too? – Michael Staples Feb 22 '20 at 14:11
  • 1
    OK so i was not sure about the framework but as i noticed that in my browser my connection is handshaking on TLS 1.3 . And yes you are correct in that case it will be an issue with local too.. – Soumen Mukherjee Feb 22 '20 at 14:16
  • Any Luck... I am interested to know what might have caused this...? – Soumen Mukherjee Feb 24 '20 at 18:16
  • No, unfortunately not. I have ebay and Microsoft support on this one now too. Waiting to find a solution and then will update this. Possibly, it isn't even an issue with the HttpClient or .NET Framework but underlying communication between Azure and ebay. – Michael Staples Feb 25 '20 at 11:57
  • 1
    That's very interesting...u have made two big boys do some extra work... – Soumen Mukherjee Feb 25 '20 at 14:21
  • Problem solved. ebay apparently had issues. But reconstruction was complicated, resulting in every party checking their side in depth. I wanted to narrow down if perhaps somebody else has had similar issues using the HttpClient. Thank you very much for helping out @SoumenMukherjee. – Michael Staples Feb 26 '20 at 08:14

1 Answers1

1

In terms to narrow down possible causes, I had ebay and Microsoft Support to check their systems. It turned out to actually be an issue on side of ebay.

Michael Staples
  • 537
  • 7
  • 13