PCF Container TrustManager adds certificates, but TrustManagerFactory loads certificates twice

Question

I run a spring boot app on PCF with an apache-httpclient 4.x. The client creates a ssl context:

        final SSLContext sslcontext = SSLContext.getInstance(algorithm);
        sslcontext.init(keymanagers, trustmanagers, params.getSecureRandom());

I get the trustmanagers as follows:

        final TrustManagerFactory tmfactory =
              TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmfactory.init((KeyStore) null);

I log the number of certificates it finds and I log the found CN's. However I found it list every certificate twice!

If I provide a keystore with 1 certificate:

 tmfactory.init((KeyStore) truststore);

It will log that certificate + all environment certificates. As follows:

   private static void logAcceptedIssuers(TrustManager[] trustmanagers) {
        log.info("Adding the following trusted certificates to the SSL context: ");
        Arrays.stream(trustmanagers)
                .filter(X509TrustManagerWrapper.class::isInstance)
                .map(X509TrustManagerWrapper.class::cast)
                .map(X509TrustManagerWrapper::getAcceptedIssuers)
                .forEach(SSLContextFactory::logAcceptedIssuers);
    }

    private static void logAcceptedIssuers(final X509Certificate[] certificates) {
        final int certificatesCount = certificates.length;
        final String prefix = "Trusted certificates (total=" + certificatesCount + "): \n";
        final String certDNs = Arrays.stream(certificates)
                .map(X509Certificate::getSubjectDN)
                .map(Principal::getName)
                .map(SSLContextFactory::extractCommonName)
                .collect(Collectors.joining(" |#| ", prefix, "\n"));

        log.info(certDNs);
    }

    @VisibleForTesting
    static String extractCommonName(String principalName) {
        ... Some code for extracting commonname from principal name...
    return cn;
    }

Where does the TrustManagerFactory find the pcf trusted certificates?

How can I check if those pcf certificates are available & loaded, or where can I get the pcf-certificates-only TrustManager. My worry is that it might cause issues if I load it twice, but I have no indication it is causing issues (but now I have 288 instead of 144 certificates in my SSLContext, does that impact performance? Can it cause issues?)).

Regards,

Rick

Answer 1

The Java buildpack adds a component called the Container Security Provider. This is what adds the functionality to automatically load Platform/Bosh trusted CA certificates, as well as the container instance ID cert and key.

To ensure platform/Bosh trusted CA certificates are trusted, the Container Security Provider adds a TrustManagerFactory. This looks at the file /etc/ssl/certs/ca-certificates.crt and trusts everything in that file.

This is in addition to the default behavior of the JVM.

I log the number of certificates it finds and I log the found CN's. However I found it list every certificate twice!

You didn't share the code to show how you're doing this, so I can only speculate. My suspicion is that you're seeing duplicates because the Container Security Provider does not replace the default behavior for trusting CA certs in the JVM, it adds an additional TrustManager.

Pair that with the fact that there is probably a lot of overlap between what's in /etc/ssl/certs/ca-certificates.crt and what's in the JVM's default trust store. I guess I could see there being duplicates.

If I provide a keystore with 1 certificate: It will log that certificate + all environment certificates.

This further's my suspicion because when you override the default JVM truststore, it sounds like the duplicates go away. That would mean you're left with your custom truststore plus what's added by the CSP.

Where does the TrustManagerFactory find the pcf trusted certificates.

https://github.com/cloudfoundry/java-buildpack-security-provider/blob/main/src/main/java/org/cloudfoundry/security/CloudFoundryContainerTrustManagerFactory.java#L41

How can I check if those pcf certificates are available & loaded, or where can I get the pcf-certificates-only TrustManager. I want this to prevent loading the certificates twice.

Is there a larger problem here? I get what you're saying about seeing certificates listed twice, but is that actually causing a problem with your application? If so, please update your question and explain what is happening? That might help to provide some more context.

Aside from that:

1. You can disable the CSP. Set and env variable JBP_CONFIG_CONTAINER_SECURITY_PROVIDER='{trust_manager_enabled: false}'.

   https://github.com/cloudfoundry/java-buildpack/blob/main/docs/framework-container_security_provider.md#security-provider

   Do understand that by disabling this, your application will not automatically trust platform/Bosh deployed CA certificates. This could cause failures. For example, if your application is binding to a CF marketplace service that is using certificates signed by a CA that is distributed through the platform/Bosh trusted CA certificates.

2. You could do the opposite & set the default Java truststore to point to an empty keystore. You'd need to create an empty keystore, probably under your application, then set -Djavax.net.ssl.keyStore=/home/vcap/app/empty.jks (/home/vcap/app is the root of your application, change the rest to point to where you store the empty keystore file).

   https://stackoverflow.com/a/2138670/1585136

If my suspicion is true, either of those would result in the duplicates going away.