0

Similar questions have been asked multiple times before - but before flagging this one as duplicate please read on. Most of these questions are very old. I have worked through a lot of questions and answers and did not find a suitable solution.

We have an Azure Cloud Service project in .net 4.5. It connects to dozens of our customers' APIs (not necessarily cloud hosted) without any problems, but a single API fails with this error message:

The request was aborted: Could not create SSL/TLS secure channel

What am I missing here?

This is the code (slightly condensed) that I am using to connect to the API (this runs per API, so the base URL does not change):

ServicePointManager.ServerCertificateValidationCallback += ValidateRemoteCertificate;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol =
    SecurityProtocolType.Tls12 |
    SecurityProtocolType.Tls11 |
    SecurityProtocolType.Tls |
    SecurityProtocolType.Ssl3;

ApiClient = HttpClientFactory.Create();
ApiClient.DefaultRequestHeaders.Authorization = null;
ApiClient.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", "Basic {passwordToken}");
ApiClient.DefaultRequestHeaders.Accept.Clear();
ApiClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

SemaphoreSlim throttler = new SemaphoreSlim(initialCount: 50);

var sp = ServicePointManager.FindServicePoint(new Uri(baseUrl));
sp.SetTcpKeepAlive(true, 30000, 30000);

foreach (var request in urls)
{
    Result = new HttpResponseMessage();
    Result = await ApiClient.GetAsync(url);
    ...
}

This is what makes it hard to debug:

  • This problem only occurs in production, i.e. when running as an Azure Cloud Service. Not when debugging locally.
  • It only occurs with requests sent through HttpClient. Not with WebClient.
  • Further research (comparing the APIs) revealed that this API is the only one that has enabled SNI and ONLY supports TLS1.2.

Suggestions considered from other questions/answers regarding SNI in .net Framework:

  • To prevent misunderstandings: This is about the cloud service connecting to an API, not about a connection that is being made to the cloud service.
  • The HttpClient instance is being reused for all requests to a single API. (This is important as this answer suggests that the SNI tag will be created with the domain HttpClient has been initialized with). I have also tried configuring TLS after the Factory instantiated the HttpClient. No change.
  • The certificates are valid of course. No self-signed certificates but regular trusted ones off the shelf. Opening the API in any browser also works like a charm.
  • TLS1.2 is not enabled by default in .net framework 4.5, but the line ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12should actually enable it. Is there anything wrong with the way I am doing it?
  • Calling the API in curl (WSL and Azure remote bash) with curl --user 'user:pwd' https://myurl also works perfectly and returns the expected data.
  • Testing tls1.2 with openssl like openssl s_client -connect hostname:443 -tls1_2 does not reveal any issues. The chain is displayed correctly, and a TLSv1.2 session is confirmed. Testing the server's SNI feature with openssl with openssl s_client -connect host:443 -tls1_2 -servername host -tlsextdebug -msg reveals SNI support by returning TLS server extension "server name" (id=0), len=0 I get the same certificate if I provide a completely different fantasy hostname though.
  • I captured the TLS/SNI handshake when debugging locally (see screenshot below). No issues. My ability to debug ends with the cloud service. I would love to see the handshake between the cloud service and the API in WireShark, but I don't know of any option to analyze network traffic at that layer on an Azure cloud service. But if anyone knows how to capture the handshake process, I'd appreciate some hints.
  • The server selects ECDHE-RSA-AES256-GCM-SHA384 as a cipher suite during the handshake with openssl which is pretty much default for TLS1.2. I don't have access to the cloud services list of cipher suites that it would provide at Client Hello - any idea how to find out?
  • I don't have any proof that SNI is actually causing the problem, but this is the only difference between this API and dozens of others I can spot.

TLS/SNI handshake captured locally in WireShark

Stack trace:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar) --- End of inner exception stack trace --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at WCFServiceWebRole1.Controllers... in [the line calling GetAsync()]

Aileron79
  • 523
  • 5
  • 27
  • Every thing changed in June this year due to Microsoft doing a security update on servers that disabled TLS 1.0/1.1. Microsoft when they do a Net update also changes to latest default values so version of SHA will also change. The SHA has to match certificate. I would check the certificate to see version of SHA. Also change code to use TLS 1.2/1.3 only : SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13 – jdweng Oct 23 '20 at 10:06
  • Thanks for highlighting this. I cannot reduce support to TLS 1.2 and 1.3 as many of the clients do not support these yet. These environments are not under my control. – Aileron79 Oct 23 '20 at 14:06
  • I think you need to discuss this with your clients. As soon as they install the microsoft security updates the app may stop working. Check the following wiki site. Make your certificate compatible with all versions of TLS that your customers are using (https://en.wikipedia.org/wiki/Transport_Layer_Security). I suspect all are using TLS 1.2/1.3 since the microsoft security update disabled the other versions of TLS – jdweng Oct 23 '20 at 14:15
  • Add TLS1.3 to your selection list. – jdweng Oct 23 '20 at 14:21
  • `this API is the only one` actually, everyone requires TLS1.2 these days. All cloud providers, major services, the payment industry etc. This means Google requires TLS1.2 too. – Panagiotis Kanavos Oct 23 '20 at 14:26
  • There definitely will be discussions with the clients. However, it is not my certificate, it is their certificates. And it is roughly 250 different customers, running their APIs on completely different platforms. While you are perfectly right, it is not realistic to have them upgrade their systems quickly. The problem in this case was actually caused by the fact that there were no matching cipher suites between client and API. So here the client needs to act as I don't have any power over the cipher suites supported by Azure Cloud Services. Thanks again! – Aileron79 Oct 23 '20 at 14:26
  • @Aileron79 tell them that Azure, AWS and Google don't support anything less than TLS1.2 for **years**. Airlines and payment services started dropping support for older versions 4-5 years ago. That's what persuaded some companies to finally upgrade from the (now) unsupported .NET 4 or 4.5 to 4.6.2, the first version that used TLS1.2 automatically – Panagiotis Kanavos Oct 23 '20 at 14:28
  • @Aileron79 specifically, the airline industry (GDSs) dropped support for TLS1.1 by December 2016. Which caused laggards to upgrade in haste. All the concerns about breaking changes from one .NET 4.x version to the next suddenly disappeared and all apps were switched instantaneously – Panagiotis Kanavos Oct 23 '20 at 14:30
  • `I don't have any power over the cipher suites supported by Azure Cloud Services`. That would be TLS1.2 and above. So don't use older ones. `it is roughly 250 different customers` there are a lot more than 250 online travel agents. – Panagiotis Kanavos Oct 23 '20 at 14:31
  • @PanagiotisKanavos: I did not check all the clients' APIs, but this was indeed the only one ONLY supporting TLS 1.2. (Which is not too bad from a security perspective of course.) There might be more, but of those I checked, there were no similar clients. I know for a fact that some of them use very old platforms. But in the end it's their data, so they are responsible for protecting it properly. We will, however, address this issue with our customers. (The API runs on different 3rd party software, so this is not really in our area of influence). – Aileron79 Oct 23 '20 at 14:33
  • @Aileron79 just because the customers pretend the earth is flat doesn't mean it is. Or that they believe it themselves. [PCI demands TLS 1.1 and strongly suggests using 1.2](https://developer.sabre.com/blog/pci-mandated-upgrade-tls-v1-2) since 2018, so those customers are **already using TLS 1.2**. Unless they don't process any payments. Or somehow escaped compliance audits. It's quite possible that if you tell them to `Can you upgrade please, like you did for PCI?` they'll answer `Sure, we forgot about it` – Panagiotis Kanavos Oct 23 '20 at 14:36
  • @PanagiotisKanavos: If we just turn off the older ones, I'll be unable to provide our services to our customers. It is them not providing TLS1.2 on their servers. We cannot force them to, we can only recommend enabling TLS1.2. And I think there is a misunderstanding: TLS1.2 does not mean that all clients/servers necessarily use the same cipher suites. If you go to https://ssllabs.com/ssltest and run a test against different TLS1.2 servers, you will see that different servers support different cipher suites - which actually was the problem in this case. – Aileron79 Oct 23 '20 at 14:38
  • Blame the entire issue on Microsoft. Yes the industry decided five years ago to eliminate TLS 1.0/1.1 and let everyone have five year to change. But the security update Microsoft pushed in June disabled TLS 1.0/1.1 on servers and did nothing to get clients to work. Here again Microsoft just changes Net Library (or windows) and does not consider forward/backwards compatibility. – jdweng Oct 23 '20 at 20:49

1 Answers1

0

I ended up re-creating a simple API on one of my servers and configured the software in such a way to send its requests there. That way I could capture the TLS handshake and analyze it in Wireshark. These are the supported cipher suites (client side, which is Azure cloud service):

Supported cipher suites on Azure cloud service

And these are the cipher suites supported by the API which is not working:

Supported cipher suites on API server

I would assume that there should be a match so that server and client can agree on one. However, I cannot find a match... guess that is what is causing the problem. In fact, the list of supported cipher suites is much longer in a local debugging session - and there is at least one match which explains why it works locally.

Aileron79
  • 523
  • 5
  • 27
  • You explicitly enabled *weaker* TLS versions in your code. All cloud venors, major services, airlines, payment processors have already switched to **at least** TLS1.2 for 4 years now. Upgrade your .NET version to one that uses TLS1.2 automatically and the problem will go away. Realistically, that's 4.7.2 or later, to avoid compatibility issues with .NET Standard packages – Panagiotis Kanavos Oct 23 '20 at 14:26