SSL/TLS connections not working when deployed as Azure App Service

Question

My team has an application that is currently deployed as an azure cloud service. The application works well, but after deploying as an app service (as a continuous web job), we're seeing many types of TLS connection failures. The TLS certificates are loaded into HTTPS clients and TCP socket clients. Why are these breaking when running as an app service?

TCP:

System.ComponentModel.Win32Exception (0x80004005): The credentials supplied to the package were not recognized
System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package, CredentialUse intent, SecureCredential scc)
System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage, SecureCredential& secureCredential)
System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output)
System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count)
System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)

HTTP:

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
System.Net.ConnectStream.WriteHeaders(Boolean async)   --- End of inner exception stack trace ---
System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
System.Net.HttpWebRequest.GetRequestStream()

Remote Certificate:

So, what's the remote certificate? Can it be validated with the stock Windows Server 2016 Root CAs (run `Get-Childitem cert:LocalMachine\root -Recurse` in Kudu)? — evilSnobu, Dec 30 '17 at 14:20
I added the remote certificate above. From a kudu powershell I can run `$cert.Verify()` and `Test-Certificate -Cert $cert`. Both of these are successful — B. McCartney, Jan 03 '18 at 23:53

evilSnobu · Answer 1 · 2018-01-04T19:50:53.760

1

The thawte Primary Root CA - G3 certificate, with thumbprintF18B538D1BE903B6A6F056435B171589CAF36BF2, is not present in the Trusted Root store on the App Service worker, thus remote cert validation fails. The leaf cert (intermediate) is missing as well.

Which doesn't make for terribly good news. I see two options, either build a chain on-the-fly (which loads the missing CA cert from disk), or blindly return true from WebRequestHandler and its ServerCertificateValidationCallback delegate property.

NOTE: I do have a working sample somewhere for the first part, let me know if i should look that up. In essence:

Load the missing root CA from disk into X509Certificate2
Build a chain and add it in
Pass it to ServerCertificateValidationCallback in WebRequestHandler

edited Jan 04 '18 at 19:50

answered Jan 04 '18 at 07:58

evilSnobu

24,582
8
41
71

You should probably open a support case with Azure Support to add both the root CA and intermediate - i can see this cert comes bundled with Windows 10, so it must be a popular issuer. – evilSnobu Jan 04 '18 at 19:52
we're moving forward with a support ticket, thanks a ton for the advice! – B. McCartney Jan 04 '18 at 23:54
It turns out we needed to add `WEBSITE_LOAD_CERTIFICATES = *` in the app settings on the portal, even though we're not loading our certificates into the portal's SSL Certificates area. If this configuration is not set, the **client certificate** will fail validation. So the remote cert never had a chance to evaluate. This was also evident by the fact that the ServerCertificateValidationCallback function was not being invoked. – B. McCartney Jan 11 '18 at 22:15
I think root certs are being downloaded as part of the handshake process. I tried reproducing the problem on my local machine by removing the thawte certs. But when I would run my sample the thawte cert would reinstall itself. – B. McCartney Jan 11 '18 at 22:25
Root CAs are never downloaded as part of the TLS handshake, only leaf certs (intermediates) and only by clients that support AIA (Authority Information Access extension) - in practice this means just browsers. Think about it, if they also push you the root cert, they may as well self sign it or issue it through Joe's Crab Shack Certificate Authority.. it would still be valid. – evilSnobu Jan 12 '18 at 01:09
Right, so they're probably not being downloaded, but the root CA does reappear into my store simply by running my sample exe. Because this causes speculation that the cert's presence in the store and it's trust are not exactly 1-to-1, I couldn't be 100% confident that searching for the root cert within the store would explain why the connection does not work – B. McCartney Jan 12 '18 at 19:27

score 1 · Accepted Answer · answered Jan 11 '18 at 22:44

1

Add this to the App Settings in the portal for your app service

WEBSITE_LOAD_CERTIFICATES = *

You still need this even if you're not loading certs in the SSL Certificates area of the portal
This will impact the web jobs for that app service

answered Jan 11 '18 at 22:44

B. McCartney

132
7

1

That doesn't explain it, since that app setting pulls the certs you upload via the portal into the Personal store ("My" store) on the web worker, it has nothing to do with Trusted Root store. But if that makes it work then there's undocumented functionality in play. – evilSnobu Jan 12 '18 at 00:53
Can you check what `Get-Childitem cert:LocalMachine\root -Recurse` returns now? Do you see thawte's thumbprint? – evilSnobu Jan 12 '18 at 00:55
During testing, the service was actually connected to a proxy server, which was causing an additional problem with the handshake. I continued testing with a different service not using the proxy. – B. McCartney Jan 12 '18 at 18:23
For my other service, the root CA was visible in the store and produced the same (TCP)exception. Like the other services we’re having a problem with, the client certificate is fetched from our repository (not through the portal). After AuthenticateAsClient is invoked, the exception appears. The ServerCertificateValidationCallback does not get invoked, but if looking at the [networking trace log](https://learn.microsoft.com/en-us/dotnet/framework/network-programming/how-to-configure-network-tracing) it does appear to be evaluating the remote certificate. – B. McCartney Jan 12 '18 at 18:26

SSL/TLS connections not working when deployed as Azure App Service

2 Answers2