Why does a C# Google.Cloud.Firestore library fail to connect with an enterprise MITM firewall?

Question

Specific error message from the library

CompletedListGrpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="Connect Failed")

• I'm particularly referring to this library: https://github.com/googleapis/google-cloud-dotnet/tree/master/apis/Google.Cloud.Firestore
• I'm having an issue with the google-cloud-firestore service particularly (although the issue might be occurring in another dependant library, perhaps authentication)
• The firewall in question is Palo Alto Networks. It has an enterprise certificate that's installed on enterprise workstations and trusted. The firewall essentially acts as a MITM to decrypt traffic for deeper analysis of TLS traffic.

When the firewall MITM feature is disabled for the firestore.googleapis.com the library works ok. When the MITM feature is enabled, it doesn't work.

Sub-questions:

1) Does the library code have a hardcoded certificate check? (I couldn't find one)

see https://github.com/googleapis/google-cloud-dotnet/blob/master/apis/Google.Cloud.Firestore.V1/Google.Cloud.Firestore.V1/FirestoreClient.cs#L550 Line ~550

public static gaxgrpc::ServiceEndpoint DefaultEndpoint { get; } = new gaxgrpc::ServiceEndpoint("firestore.googleapis.com", 443);

• It doesn't make sense to hardcode the certificate, because certificates are renewed, and sometimes old ones are revoked.
• grpc is quite standard and shared beyond use by Google (See https://grpc.io/).
• It's reasonable to assume grpc doesn't include an explicit certificate for a downstream dependency
• If firestore was going to assert a certificate, it's reasonable to assume that would be done on this line of code along with the URL

2a) Does .Net Framework automatically trust certificates that are in the windows trust store? Is any code required to make this work?

It appears that .Net Framework works with Windows Certificate Store - see https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/working-with-certificates

2b) Perhaps the certificate is only trusted but the interactive user, and not the whole machine, and therefore service accounts don't see that certificate - I'll check this...

3) Will the library have a specific error about the certificate if that's the cause of a "Connect Failed"?

Answer 1

It's likely to be an issue between GRPC and Windows Certificate store. Whether that's a fundamental issue with GRPC (rather than .Net) or due to running it under a different account, hardcoding the correct Root CA for your firewall into the application will certainly give you total control.

You might be able to override the DefaultEndpoint/Channel to use your own ServicePoint that also includes a hardcoded certificate

see TLS support for GRPC in C#

  var cacert = File.ReadAllText(@"../ca.crt");
  var clientcert = File.ReadAllText(@"../client.crt");
  var clientkey = File.ReadAllText(@"../client.key");
  var ssl = new SslCredentials(cacert, new KeyCertificatePair(clientcert, clientkey));
  var channel = new Channel("firestore.googleapis.com", 443, ssl);
  //etc..

Create your own endpoint, then supply it explicitly so that it overrides the DefaultEndpoint. Here's the function you would call with your Channel. (see https://github.com/googleapis/google-cloud-dotnet/blob/master/apis/Google.Cloud.Firestore.V1/Google.Cloud.Firestore.V1/FirestoreClient.cs#L550)

public static FirestoreClient Create(grpccore::Channel channel, FirestoreSettings settings = null)
{
    gax::GaxPreconditions.CheckNotNull(channel, nameof(channel));
    return Create(new grpccore::DefaultCallInvoker(channel), settings);
}

Also, have a good look at the SslCredentialsTest code - https://github.com/grpc/grpc/blob/master/src/csharp/Grpc.IntegrationTesting/SslCredentialsTest.cs

Note: Usually a channel will be created with the s_channelPool. If you create it manually it might not use the pool - there might not be a way to do that. That may or may not be a problem for you.

---

You may be able to extend this to manually read the public certificate information from the Windows Certificate Store at runtime by name, selecting the most recent. This would mean that you can use app.config to assign the CN of the certificate to get from the Windows Certificate Store, then you won't need to manually send updated versions of your software.

---

This link seems to indicate the by default "Publically Trusted Roots" are trusted: https://forum.predix.io/questions/30875/event-hub-c-client-how-to-deal-with-the-tls-certif.html

var channel = new Channel("greeter.googleapis.com", new SslCredentials());  // Use publicly trusted roots.

This means that although there are no "hardcoded" specific certificates for firestore, it doesn't normally integrate with Windows Certificate Store, and therefore any custom installed Root CAs are not trusted.

Looking more closely at the library source code [https://github.com/googleapis/gax-dotnet/blob/master/Google.Api.Gax.Grpc/ChannelPool.cs]:

public Channel GetChannel(ServiceEndpoint endpoint, IEnumerable<ChannelOption> channelOptions)
{
    GaxPreconditions.CheckNotNull(endpoint, nameof(endpoint));
    var credentials = _lazyScopedDefaultChannelCredentials.Value.ResultWithUnwrappedExceptions();
    return GetChannel(endpoint, channelOptions, credentials);
}

The credentials parameter is filled automatically, and following _lazyScopedDefaultChannelCredentials, GoogleCredential.GetApplicationDefaultAsync(), we end up at CreateDefaultCredentialAsync which refers to your Google Services file which you likely downloaded from firebase somewhere.

see https://github.com/googleapis/google-api-dotnet-client/blob/master/Src/Support/Google.Apis.Auth/OAuth2/DefaultCredentialProvider.cs

Note:

There a good chance this answer only leads to custom client-side credentials, rather than the ability to have hardcoded credentials for a custom server-side Root CA.

Answer 2

It might simply be a misconfiguration with Windows Certificate store. Perhaps the account you run the software on doesn't have access to the correct Root CA. Whatever the reason, there is a common .Net way to manually approve a certificate.

1) First, try a test which simply bypasses checks. see https://stackoverflow.com/a/2675183/887092

ServicePointManager
    .ServerCertificateValidationCallback += 
    (sender, cert, chain, sslPolicyErrors) => true;

Don't let this code remain in production, only use it to see if this pathway can work.

2) Next customise this function to manually check for the specific certificate(s) for the Root CA and associated certificates. It's highly recommended to use a library to help you, but you can probably make it work with core .net libraries.

see C# / .NET - How to allow a "custom" Root-CA for HTTPS in my application (only)?

Conclusion

If this answer doesn't work, I will delete it.