0

I am trying to port this python web client to C# .NET Core 3.1. I'm running on Windows 10x64 (build 19041). The linked github project contains a (secure) test web server sandbox you can develop against together with some client and server test self-cert certificates.

I am not very familiar with python (or SSL/TLS for that matter), but I can

  1. run the test suites which are included in the project - which involved starting and stopping the test server included.
  2. start up the test server and then cUrl it successfully with the included client certificates/keys/etc (so long as I instruct cUrl not to check certificate revocation)

If I run a python utility (sslyze) against the test server it indicates that it supports TLS1.3.

I have

  1. imported the client.cert.pem into my personal certificate store and the ca.cert.pem into the trusted root certificate authority. (I wasn't sure if this was correct and perhaps I needed to unpack them, but they appear to have been imported successfully)

  2. Added the following registry keys

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client] "DisabledByDefault"=dword:00000000 "Enabled"=dword:00000001

When I try and run the C# code, I get the

"IOException: Unable to read data from the transport connection"

error on the response.Result line.

I'm struggling to make any headway, now.

I'd be really grateful for any pointers as to where to look next. Many thx IA.

Here's my code

class Program
{
    
    private const string Thumbprint = "37af35e3f13c65ea0d7bbad148332924a1ce41d7";
    private const string MailBox = "alice";
    private const int Port = 8000;
    private const string BaseAddress = "127.0.0.1";
    
    private static HttpClient _client = null;

    public static void Main(string[] args)
    { 
        ConfigureHttpClient();
        var response = Handshake();

        Console.WriteLine($"Handshake succeeded: {response.StatusCode == HttpStatusCode.OK}");           
    }

    private static HttpResponseMessage Handshake()
    {
        var uri = new Uri($"https://{BaseAddress}:{Port}/messageexchange/{MailBox}");
        var response = _client.PostAsync(uri, null);            
        return response.Result;
    }

    private static void ConfigureHttpClient()
    {
        var handler = new HttpClientHandler
        {
            ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; },                 
            CheckCertificateRevocationList = false,                
            SslProtocols = SslProtocols.Tls13
                           | SslProtocols.Tls12
                           | SslProtocols.Tls11
                           | SslProtocols.Tls
        };
        var certificate = GetCertificateByThumbprint(Thumbprint);
        handler.ClientCertificates.Add(certificate);
        _client = new HttpClient(handler);                      
    }

    //Returns a certificate by searching through all likely places
    private static X509Certificate2 GetCertificateByThumbprint(string thumbprint)
    {
        X509Certificate2 certificate;
        //foreach likely certificate store name
        foreach (var name in new[] { StoreName.My, StoreName.Root })
        {
            //foreach store location
            foreach (var location in new[] { StoreLocation.CurrentUser, StoreLocation.LocalMachine })
            {
                //see if the certificate is in this store name and location
                certificate = FindThumbprintInStore(thumbprint, name, location);
                if (certificate != null)
                {
                    //return the resulting certificate
                    return certificate;
                }
            }
        }
        //certificate was not found
        throw new Exception(string.Format("The certificate with thumbprint {0} was not found", thumbprint));
    }

    private static X509Certificate2 FindThumbprintInStore(string thumbprint,
                                                          StoreName name, StoreLocation location)
    {
        //creates the store based on the input name and location e.g. name=My
        var certStore = new X509Store(name, location);
        certStore.Open(OpenFlags.ReadOnly);
        //finds the certificate in question in this store
        var certCollection = certStore.Certificates.Find(X509FindType.FindByThumbprint,
                                                         thumbprint, false);
        certStore.Close();

        if (certCollection.Count > 0)
        {
            //if it is found return
            return certCollection[0];
        }
        else
        {
            //if the certificate was not found return null
            return null;
        }
    }
}

Here's the full error stack

System.AggregateException HResult=0x80131500 Message=One or more errors occurred. (An error occurred while sending the request.)
Source=System.Private.CoreLib StackTrace: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result() at IfxMeshClient.Program.Handshake() in C:\Users...\IfxMeshClient\Program.cs:line 41 at IfxMeshClient.Program.Main(String[] args) in C:\Users...\IfxMeshClient\Program.cs:line 32

Inner Exception 1: HttpRequestException: An error occurred while sending the request.

Inner Exception 2: IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine..

Inner Exception 3: SocketException: An established connection was aborted by the software in your host machine.

EDIT: I just wanted to add the output from sslyze as I wondered if it could be a cipher mismatch, but these ciphers appear to be available on my version of win10.

* TLS 1.3 Cipher Suites:
     Attempted to connect using 5 cipher suites.
     The server accepted the following 3 cipher suites:
        TLS_CHACHA20_POLY1305_SHA256                      256       ECDH: X25519 (253 bits)
        TLS_AES_256_GCM_SHA384                            256       ECDH: X25519 (253 bits)
        TLS_AES_128_GCM_SHA256                            128       ECDH: X25519 (253 bits)
     The group of cipher suites supported by the server has the following properties:
       Forward Secrecy                    OK - Supported
       Legacy RC4 Algorithm               OK - Not Supported
     The server is configured to prefer the following cipher suite:
        TLS_AES_256_GCM_SHA384                            256       ECDH: X25519 (253 bits) 

EDIT1: Not quite sure what to make of Fiddler output. I'm not sure if it confuses things since it refers to TLS12 and then suggests supported versions are TLS13

CONNECT 127.0.0.1:52985 HTTP/1.1
Host: 127.0.0.1:52985

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.3 (TLS/1.2)
Random: F7 09 8F A8 9A 72 AF 33 93 29 90 48 E4 A2 E2 09 CE BD 7E 27 94 0A C9 53 D4 54 6C 35 0B DF E2 E9
"Time": 12/08/2059 21:41:27
SessionID: empty
Extensions: 
    supported_versions  Tls1.3
    signature_algs  rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha1, ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_sha1, dsa_sha1, rsa_pkcs1_sha512, ecdsa_secp521r1_sha512
    supported_groups    x25519 [0x1d], secp256r1 [0x17], secp384r1 [0x18]
    key_share   00 24 00 1D 00 20 95 34 AC 4A A5 6C D8 24 CD 50 29 2E F8 27 B4 59 D7 2C D8 C0 79 7B 14 79 B2 E8 28 6F 36 DE D4 48
    post_handshake_auth
    extended_master_secret  empty
    renegotiation_info  00
    psk_key_exchange_modes  01 01
Ciphers: 
    [1302]  TLS_AES_256_GCM_SHA384
    [1301]  TLS_AES_128_GCM_SHA256
    ...

and the response was

HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 13:21:35.087
Connection: close

fiddler.network.https> HTTPS handshake to 127.0.0.1 (for #6) failed. System.Security.Authentication.AuthenticationException A call to SSPI failed, see inner exception. < The message received was unexpected or badly formatted

Win32 (SChannel) Native Error Code: 0x80090326

Further the combination of the fiddler output as well as the sslyze output would seem to indicate there are common ciphers as well

TLS_AES_256_GCM_SHA384
TLS_AES_128_GCM_SHA256
Simon Woods
  • 905
  • 1
  • 6
  • 13
  • The certificate validation is done by performing a virtual http connection so a connection has to be completed. The connection is made using TLS. Micrsoft in Jume pushed a security update that disabled TLS 1.0/1.1 on servers but did nothing with the client. The client selects the TLS version as part of the request. So one possibility is the TLS version is wrong. I would add before the validation the following : ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12; – jdweng Nov 13 '20 at 11:17
  • Thx. ... ah. I understood that with .NET Core that had been added to the HttpClientHandler as SslProtocols. Is that not the case? – Simon Woods Nov 13 '20 at 11:25
  • Having said that though, I've tried removing the ref to tls1 & 1.1 and get the same behaviour. Thx again though. – Simon Woods Nov 13 '20 at 11:33
  • TLS is a SSL Protocol. SSL is older version of TLS and isn't used much any more. But the parameter is still called SSL. – jdweng Nov 13 '20 at 11:54
  • Are you specifying the TLS version as 1.2? If not the default version is used which still may be 1.0 or 1.1. The certificate also must be an encryption mode supported by the TLS version (see Wiki : https://en.wikipedia.org/wiki/Transport_Layer_Security). the server may want TLS 1.3 so you may want either 1.2 or 1.3 : SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13 – jdweng Nov 13 '20 at 12:01
  • Yes. I updated it to specify 1.2 | 1.3 and removed ref to the older versions. Same problem unfortunately. (FYI I have a .NetCore and a .Net4.8 synchronous ver - to see if the difference in errors reveals anything. It doesn't - unsurprisingly!!) – Simon Woods Nov 13 '20 at 12:04
  • How long does it take for failure to occur. If it is 30 seconds than you are looking for a proxy and need to set proxy to null. The TLS should be trying all the encryption modes and find which one works. The certificate needs to be on both client and server and needs to be in both the user and machine registers. You do have some old encryption modes that may be causing an exception. Did you check in the Event Viewer on client and server to see if there are any error message? – jdweng Nov 13 '20 at 12:10
  • It fails pretty instantly. There are no errors in the windows system event log. I've rechecked the LM and CU certificates and I can see the cert in the personal folder and the ca in the trusted root ca folder in both areas. – Simon Woods Nov 13 '20 at 12:25
  • The server uses pem files for its tls stuff. Hence when I run the server and fire cUrl at it with the appropriate cert files, it returns success. Thus istm that it is something on the windows side of tls which I've not configured correctly or something – Simon Woods Nov 13 '20 at 12:32
  • I would step through code and check FindThumbprintInStore if it is finding the certificate. You can also try true and see if it automatically finds the certificate : CheckCertificateRevocationList = false, – jdweng Nov 13 '20 at 13:11
  • Seems to be finding it ok. Setting CheckCertificationRevocationList (to either true or false) doesn't seem to make any difference. – Simon Woods Nov 13 '20 at 13:20
  • Are you running the python on same machine as the c# code? TLS is handle by the operating system in c#. I'm not sure if the python is using the operating system or has it own handler for TLS. So I'm suspecting the error is in the phone or the kernel under the phone. Both need to be the latest that supports TLS 1.3. I'm looking at your latest changes and TLS there are retires. A negotiation occurs between the client and server to determine a compatible mode and the retries are trying to find a common mode. Comparing with fiddler the python and c# will help. – jdweng Nov 13 '20 at 13:46
  • Looking at Fiddler TLS you will see a certificate block where the server sends a list of certificate names. The client looks up the names. I think that is done in your callback method. – jdweng Nov 13 '20 at 13:48
  • You are getting a HTTP response with 200. This means the TLS completed. The TLS occurs before the HTTP request. You are getting a message : "The message received was unexpected or badly formatted". So you have to look at the body of the 200 response and see what the data looks like. I can think of 3 reasons for failure 1) The data is not getting decrypted 2) You are getting wrong format like Xml vs JSON 3) The data may be GZIP and you have to uncompress. Fiddler will automatically decompress so you need to look at header in response to see if GZIP was used. – jdweng Nov 13 '20 at 13:55
  • ok ... so if I include fiddler, the callback doesn't get hit. However, if I remove it, it does but there is an sslPolicyError RemoteCertificateNameMismatch. I was just returning true thus ignoring the error. i presume that can't be safe? – Simon Woods Nov 13 '20 at 13:56
  • Fiddler must be getting the Certificate from the Stores and bypassing your code. That also indicates the hears in the request are good and you are getting the correct response. With fiddler do you get the correct data? It looks like the callback is bad and you have to step through can see what certificate names you are using. Something must be wrong with the parameters in the callback. You must do this without Fiddler. You know what certificate names is being used with fiddler. You should be using the same. – jdweng Nov 13 '20 at 14:17
  • So the callback error was on account of the certificate requiring "localhost" as servername and I had the server name "127.0.0.1". It went away once I changed to localhost. I still get the "The message received was unexpected or badly formatted" in fiddler as a response – Simon Woods Nov 13 '20 at 15:48
  • So you got a 200 OK with a message indicating "badly formatted" in the body? And you get a good data with python? This would indicate there is something different in the request between python and c#. You would not get a response if the TLS was bad. The default headers in c# are not the same a python. Maybe the User Agent is different which is a common issue. – jdweng Nov 13 '20 at 15:54
  • @jdweng Many, many thx for your help. – Simon Woods Nov 13 '20 at 17:42

1 Answers1

0

Just to say that, my issue appears to have have been that although I had imported the pem into the certificate store, I had not created a pfx which incorporated the private key file.

See this SO post - Convert a CERT/PEM certificate to a PFX certificate

Having done that I did get more response from the server but was still getting an Http 403 and the underlying error was

Cannot determine the frame size or a corrupted frame was received

According to this post .NET 4.8 TLS 1.3 Issue on Windows 10, it may be an ongoing issue.

Simon Woods
  • 905
  • 1
  • 6
  • 13
  • I don't believe the 2nd link. The TLS is handled in the operating system and works with python (and fiddler). So TLS 1.3 is working. You cannot create certificates for TLS 1.3 in Net, but you can connect with Net. Why are you using the Callback? The Server sends a list of Certificate Names. If you just accept what the server sends everything should work. The callback is needed only if you want to use a specific certificate and not all the ones the server accepts. Are you installing the same certificate in server and client? – jdweng Nov 13 '20 at 17:54
  • thx vm once again. Wrt the CallBack, I am just returning true irrespective. I think that python has its own implementation of tls1.3. The Python server references a different certificate to the cert I am installing in the store and using in my client code. This link also seems to suggest that it's bleeding edge for windows (or .net at least) https://stackoverflow.com/questions/64591531/c-sharp-tls1-3-exception-cannot-determine-the-frame-size-or-a-corrupted-frame-w?noredirect=1&lq=1 – Simon Woods Nov 16 '20 at 08:58
  • TLS is handles in the operating system and not in either Net or Python. The default version when you do not specify the version has a lot of different factors. In c# you can specify the version like this : System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls13 | SecurityProtocolType.Tls12; The same certificate that is used in Python is also used in c#. The certificate has to be stored for both the machine and the users. – jdweng Nov 16 '20 at 09:13
  • OK So, it now works ok with tls1.2 (having corrected the authorization code, and incorporating the private key with the cert). If I offer 1.3 as an option, I still get the "Cannot determine the frame size or a corrupted frame was received". – Simon Woods Nov 16 '20 at 11:26
  • TLS is done in the operating system and the operating system need to work with TLS 1.3. If the browser works manually (or python) to a TLS 1.3 website than c# should also work. What machine are you using? You may need to update the operating system or kernel. – jdweng Nov 16 '20 at 11:35