PFX Certificate Imported for TLS/SSL Encryption of MQTTnet Client Messages Works with Service but Fails with Xamarin UWP App

Question

What We're Trying to Do

We in the process of encrypting our MQTT messages. We've been using MQTTnet. This supports TLS encryption, so it is a question of enabling this functionality for an existing solution. Our UWP Xamarin application and a .NET console application both use the same .NET standard DLL for all things MQTT. For the broker, we were using Mosquitto, but we're planning to switch to a simple application that uses MQTTnet server.

Here's the Problem

We've successfully added TLS encryption on the broker side as well as one of a console application but are having trouble with our UWP Xamarin application. In its current state it is throwing an exception with the message that says "The client certificate provided is missing the required private key information."

What We're Looking For

confirmation that this is possible
a basic understanding of why this is not working (i.e. confirmation of our assumptions or otherwise)
a solution to this problem
ideas of things to check.

So if someone knows "the answer," great. Otherwise, ideas, advice, expertise, shared experience, etc.

Our Assumptions

we're assuming that this is possible
this does not appear to be a MQTTnet bug or limitation
we're assuming that the reason that we get this error when using the Xamarin UWP application is because of some restriction to disk access or registry access
we've been assuming that because the PFX file works with the console applicaton it should also work for the Xamarin UWP application (but we've played around with which store it is in as the Xamarin UWP application can only access it in the user key store)

Things We've Tried

we've read specification docs on MSDN
we've read blog posts
we've read and followed advice from various Stack Overflow posts
we've read and followed advice from various MQTTnet support posts
we've tried two different brokers (Mosquitto and a sample application that uses MQTTnet server).
we've tried creating the certificates using OpenSSL and manually through the OS (Windows 10); for more controllable/deterministic/repeatable results we will switch to using a PS script for the latter
we've tried creating the certificate for user store and machine store
we've tried importing the certificate into the store (see code)
we've tried many different combinations for X509KeyStorage flags (i.e. exportable, persist, user key set, etc.)
we've tried running visual studio as administrator
we've uses SysInternals ProcMon and tried to determine where this is failing (i.e. HD access or registry access)
we've tried enabling different capabilities for the Xamarin application

Useful Links

Some of our Code

    /// <summary>
    /// Connect to the MQTT broker using the defined options
    /// </summary>
    private async Task ConnectAsync()
    {
        IMqttClientOptions options = CreateMqttClientOptions();
    
        try
        {
            await m_mqttClient.ConnectAsync(options);
        }
        catch (Exception ex)
        {
            m_logger?.LogCritical(ex, "Failed to reconnect - service unavailable");
        }
    }
    
    /// <summary>
    /// Helper function used to create the MQTT client options object. This includes the certificate.
    /// </summary>
    private IMqttClientOptions CreateMqttClientOptions()
    {
        string filepath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData), "MobileTools.2.pfx");
    
        X509Certificate2 certificate = new X509Certificate2(
            filepath,
            "notactuallymypassword",
            X509KeyStorageFlags.Exportable | X509KeyStorageFlags.UserKeySet);
    
        //InstallCertificate(certificate);
    
        // Set-up options.
        return new MqttClientOptionsBuilder()
            .WithCleanSession(true)
            .WithClientId(m_clientID)
            .WithTcpServer("NotActuallyDnsName", m_configuration.Port)
            .WithTls(new MqttClientOptionsBuilderTlsParameters
            {
                Certificates = new List<byte[]>
                {
                    certificate.Export(X509ContentType.Cert)
                },
                CertificateValidationCallback = (X509Certificate xCertificate, X509Chain xChain, SslPolicyErrors sslPolicyErrors, IMqttClientOptions clientOptions) =>
                {
                    return true;
                },
                UseTls = true
            })
           .Build();
    }
    
    /// <summary>
    /// Helper function used to create the MQTT client options object. This includes the certificate.
    /// </summary>
    private void InstallCertificate(X509Certificate2 certificate)
    {
        X509Store store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
    
        store.Open(OpenFlags.ReadWrite);
        store.Add(certificate);
        store.Close();
    }

Stack Trace

The client certificate provided is missing the required private key information. ---> System.ArgumentException: The parameter is incorrect.

The client certificate provided is missing the required private key information.
   at Windows.Networking.Sockets.StreamSocketControl.put_ClientCertificate(Certificate value)
   at MQTTnet.Implementations.MqttTcpChannel.ConnectAsync(CancellationToken cancellationToken)
   at MQTTnet.Internal.MqttTaskTimeout.WaitAsync(Func`2 action, TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Adapter.MqttChannelAdapter.ConnectAsync(TimeSpan timeout, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MQTTnet.Adapter.MqttChannelAdapter.WrapException(Exception exception)
   at MQTTnet.Adapter.MqttChannelAdapter.ConnectAsync(TimeSpan timeout, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ConnectAsync(IMqttClientOptions options, CancellationToken cancellationToken)
   at MQTTnet.Client.MqttClient.ConnectAsync(IMqttClientOptions options, CancellationToken cancellationToken)
   at TS.Orbit.MQTTLib.Client.MqttNetClient.ConnectAsync(IMQTTClientConfiguration configuration)

Just added a little bit more in "what we're looking for" as "the answer" is a bit vague. — John Coffin, Aug 11 '20 at 23:44
Removed a line from the code that wasn't pertinent to the problem. — John Coffin, Aug 11 '20 at 23:46
I fixed some typos and changed "service" to "console application" and .NET core DLL to .NET standard DLL. I'm not sure if these details are relevant but they may be. — John Coffin, Aug 12 '20 at 22:29
Could you provide a stack trace that displays the error "The client certificate provided is missing the required private key information"? — dsum, Aug 19 '20 at 14:57
If you run a Wireshark trace you'll see the TLS handshake and subsequent RST or Fin, question is which side is first? — JWP, Sep 01 '20 at 23:28
Neither. With the app that is failing, we do not see any Wireshark traces between the client and broker. FYI I tested the set-up with the other app.In that case we see all of the handshake and subsequent application messages in the Wireshark traces. — John Coffin, Sep 02 '20 at 01:17
In the case of the "other app" -- the one that works -- the client side sends the first message. — John Coffin, Sep 02 '20 at 20:08

score 0 · Answer 1 · edited Nov 10 '20 at 23:24

I had almost exactly the same problem, for me the solution was to create and implement a custom TLS parameters class that extends the base MqttClientOptionsBuilderTlsParameters class and overrides the Certificates property.

public class CustomTLSParameters : MqttClientOptionsBuilderTlsParameters
    {
        public new IEnumerable<X509Certificate> Certificates { get; set; }
    }

So the client ended up being:

var MQTTClient = new MqttFactory().CreateManagedMqttClient();
var url = "myIP";
var port = 8883;
var certs = new List<X509Certificate> {
       new X509Certificate2("myCert.pfx")
};
var tlsParams = new CustomTLSParameters () {
                    AllowUntrustedCertificates = true,
                    UseTls = true,
                    Certificates = certs,
                    IgnoreCertificateChainErrors = true,
                    IgnoreCertificateRevocationErrors = true
                };
var options = new ManagedMqttClientOptionsBuilder()
                    .WithAutoReconnectDelay(TimeSpan.FromSeconds(5))
                    .WithClientOptions(new MqttClientOptionsBuilder()
                        .WithClientId(Guid.NewGuid().ToString())
                        .WithTcpServer(url, port)
                        .WithTls(tlsParams)
                        .WithCleanSession()
                        .Build())
                    .Build();

await MQTTClient.StartAsync(options);

I think it has to do with the MqttClientOptionsBuilderTlsParameters class definition for UWP, I noticed it defined IEnumerable<IEnumerable<bytes>> instead of IEnumerable<X509Certificate> for the Certificates property.

*Note: MQTTnet v3.0.13