WebRequest with certificate fails on first call and succeeds all subsequent calls

Question

Infrastructure

I have an old .NET Framework application that I migrated from an old server. It's in 4.6.2 running on Windows Server 2012 R2.

The forms application makes a call to a WCF service running also in the same server under the same application pool and the WCF service makes a WebRequest call to an external service.

The external call requires certificates for the SSL/TLS handshake.

In the old server the certificates are attached to the request, but in the new server this wasn't working and we had to add them in the trust store. This might be a important clue to the question below.

The problem

The WCF service makes the call to the external service and the first call (after deploying or restarting the app) is always an error The request was aborted: Could not create SSL/TLS secure channel.

However ALL subsequent calls work successfully. It's almost as if the certificate isn't properly loaded into the user profile on the first call, but afterwards it is.

This is causing issues after deploying a new version, all first calls fail, which impacts live traffic. We have the same configuration on 6 boxes, under a load balancer and it's identical behaviour on all 6 boxes.

I want to fix this so that the call succeeds on the first call.

The code

    public static Stream SendRequest(string url, string requestXml, string clientKeyBase64, string clientKeyPassword, int requestTimeout)
    {
        Stream os = null;
        HttpWebRequest req = null;
        WebResponse resp = null;
        byte[] bytes;

        try
        {
            req = (HttpWebRequest)WebRequest.Create(url);
            req.Timeout = requestTimeout;
            req.ContentType = "application/xml; charset=utf-8";
            req.Method = "POST";
            bytes = Encoding.ASCII.GetBytes(requestXml);
            req.ContentLength = bytes.Length;

            ServicePointManager.Expect100Continue = true;
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

            if (!string.IsNullOrWhiteSpace(clientKeyBase64))
            {
                var cert = new X509Certificate2(Convert.FromBase64String(clientKeyBase64), clientKeyPassword);
                req.ClientCertificates.Add(cert);
            }
            using (os = req.GetRequestStream())
            {
                os.Write(bytes, 0, bytes.Length);
            }

            using (resp = req.GetResponse())
            {
                using (var stream = resp.GetResponseStream())
                {
                    MemoryStream memStream;
                    memStream = new MemoryStream();
                    stream.CopyTo(memStream);
                    memStream.Seek(0, SeekOrigin.Begin); // Required to reset stream pointer at the beginning
                    return memStream;
                }
            }
        }
        catch (Exception ex)
        {
            throw ex;
        }
    }

Things I have tried

This article titled “WCF Client: First call fails, second call succeeds” is certificate related and extremely promising. However, the suggestion to bypass HTTP status 100 with ServicePointManager.Expect100Continue = true; didn’t work.
Tried setting the protocol to TLS 1.2: ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Tried setting X509KeyStorageFlags.MachineKeySet as per this suggestion
From same suggestion, tried changing the application pool identity to Network Service, LocalSystem and LocalService none of which worked.

Other info

The Application pool is the recommended identity `ApplicationPoolIdentity" and the Load User Profile setting is set to true, otherwise the certificates do not load.

I'm running out of ideas what to do next and need some help understanding and fixing this behaviour

Use tools such as Wireshark to see further into the handshake failure, and with that you can know better what to do next. — Lex Li, May 07 '20 at 03:53
Since deploy new version won't update application pool, have you tried to recycle app pool after deploying or warm up with application pool initalization? It sounds like schannel was not awake in the first call.https://learn.microsoft.com/en-us/iis/get-started/whats-new-in-iis-8/iis-80-application-initialization — Jokies Ding, May 07 '20 at 07:10

score 0 · Answer 1 · answered May 07 '20 at 14:27

It seems to work now by adding these keys:

X509KeyStorageFlags.UserKeySet |                                                                                                                                 
X509KeyStorageFlags.MachineKeySet |                                                                                                                                  
X509KeyStorageFlags.PersistKeySet |                                                                                                        
X509KeyStorageFlags.Exportable

Thanks

WebRequest with certificate fails on first call and succeeds all subsequent calls

1 Answers1