2

The context is that I am working on a Kubernetes project, where we use a Geode cluster, and Spring Boot, also Spring Boot Data Geode (SBDG). We have developed an app with it, a ClientCache. Also we have a proprietary internal mechanism to generate cluster-internal certificates, this mechanism automatically renews certificates according to the best practices. We convert the PEM formatted certificate in our App code to JKS, and configured Spring with @EnableSSL annotation to take them.

So the issue is, that everything works wonderfully for the first cycle, when the connections were created with the JKS file the App initially started up with, however if the certificate is renewed, say hourly (in cloud this is best practice), Geode fails to be connected with a bunch of Exceptions, sometimes SSLException (readHandshakeRecord), many times with "Unable to connect to any locators in the list" (but I debugged, and it is also a HandshakeException, just wrapper in a connection-exception instead). The locators and servers are up and running (I checked with GFSH), just the App I think tries to connect with the old SSLContext and fails in the SSL handshake.

The only way so far I have found is to restart the App completely, but we would need this system to be automatic, and highly available, so this should not be the only way around this issue.

I think this problem is affecting a lot of Spring/Java projects as I have found this issue all around (Kafka, PGSQL, etc...).

Do any of you have any method to do this? Is there a way to:

  • Recreate all the connections without me restarting the App?
  • Invalidate the currently used connections somehow, and force the ClientCache to create new ones, re-reading the JKS file?
  • Maybe let the Client App timeout the connections and destroy them, and create new ones, with refreshed SSLContext?

I did not find any possibilities for this.

EDIT: Let me add some code, to show how we do things, since we use Spring, it is dead simple:

@Configuration
@EnableGemfireRepositories(basePackages = "...")
@EnableEntityDefinedRegions(basePackages = "...")
@ClientCacheApplication
@EnableSsl(
    truststore = "truststore.jks",
    keystore = "keystore.jks",
    truststorePassword = "pwd",
    keystorePassword = "pwd"
)
public class GeodeTls {}

And that is it! We then use normal annotations for @Regions, and @Repositories, and we have our @RestControllers where we call the repository methods, most of them are just empty ones, or default ones as we use the OQL annotate method to do things with Spring. Since Geode has a property-based config, we never set KeyStores, TrustStores, I just happen to see them inside the code during debugging.

EDIT2: I have finally solved thanks to the below comments, it was this Geode ticket that helped a lot (thanks to Jen D for that): https://github.com/apache/geode/pull/2244, available since Geode 1.8.0. Also the below snippet was extremely useful about the Swappable KeyManager (thanks for Hakan54), I did a combined solution in the end. I had to be careful though, to set the default SSLContext only once as the subsequent sets were ineffective, and did not result in any failures. Now the App is stable it seems across certificate changes.

2 Answers2

1

I came across your question yesterday and was working on a prototype. I think it might be possible in your case. However I just tried it out locally with a http client and a server which I was able to change the certificates at runtime without the need of restarting these applications or recreating the SSLContext.

Option 1

From your question I can understand that you are reading PEM files from somewhere and converting it to something else and at the end you are using a SSLContext. In that case I would assume you are creating a KeyManager and a TrustManager. If thats the case what you need to do is create a custom implementation of the KeyManager and TrustManager as a wrapper class to delegate the method calls to the actual KeyManager and TrustManager within the wrapper class. And also add a setter method to change the internal KeyManager and TrustManager when the certificates get updated.

In your case that would be a file-watcher which gets triggered when the PEM files have been changed. In that case you only need to regenerate the KeyManager and TrustManager with the new certificates and give it to the wrapped class by calling the setter method. Below is an example code snippet what you could use:

HotSwappableX509ExtendedKeyManager

import javax.net.ssl.SSLEngine;
import javax.net.ssl.X509ExtendedKeyManager;
import java.net.Socket;
import java.security.Principal;
import java.security.PrivateKey;
import java.security.cert.X509Certificate;
import java.util.Objects;

public final class HotSwappableX509ExtendedKeyManager extends X509ExtendedKeyManager {

    private X509ExtendedKeyManager keyManager;

    public HotSwappableX509ExtendedKeyManager(X509ExtendedKeyManager keyManager) {
        this.keyManager = Objects.requireNonNull(keyManager);
    }

    @Override
    public String chooseClientAlias(String[] keyType, Principal[] issuers, Socket socket) {
        return keyManager.chooseClientAlias(keyType, issuers, socket);
    }

    @Override
    public String chooseEngineClientAlias(String[] keyTypes, Principal[] issuers, SSLEngine sslEngine) {
            return keyManager.chooseEngineClientAlias(keyTypes, issuers, sslEngine);
    }

    @Override
    public String chooseServerAlias(String keyType, Principal[] issuers, Socket socket) {
            return keyManager.chooseServerAlias(keyType, issuers, socket);
    }

    @Override
    public String chooseEngineServerAlias(String keyType, Principal[] issuers, SSLEngine sslEngine) {
            return keyManager.chooseEngineServerAlias(keyType, issuers, sslEngine);
    }

    @Override
    public PrivateKey getPrivateKey(String alias) {
        return keyManager.getPrivateKey(alias);
    }

    @Override
    public X509Certificate[] getCertificateChain(String alias) {
        return keyManager.getCertificateChain(alias);
    }

    @Override
    public String[] getClientAliases(String keyType, Principal[] issuers) {
        return keyManager.getClientAliases(keyType, issuers);
    }

    @Override
    public String[] getServerAliases(String keyType, Principal[] issuers) {
        return keyManager.getServerAliases(keyType, issuers);
    }

    public void setKeyManager(X509ExtendedKeyManager keyManager) {
        this.keyManager = Objects.requireNonNull(keyManager);
    }

}

HotSwappableX509ExtendedTrustManager

import javax.net.ssl.SSLEngine;
import javax.net.ssl.X509ExtendedTrustManager;
import java.net.Socket;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.Objects;

public class HotSwappableX509ExtendedTrustManager extends X509ExtendedTrustManager {

    private X509ExtendedTrustManager trustManager;

    public HotSwappableX509ExtendedTrustManager(X509ExtendedTrustManager trustManager) {
        this.trustManager = Objects.requireNonNull(trustManager);
    }

    @Override
    public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
        trustManager.checkClientTrusted(chain, authType);
    }

    @Override
    public void checkClientTrusted(X509Certificate[] chain, String authType, Socket socket) throws CertificateException {
        trustManager.checkClientTrusted(chain, authType, socket);
    }

    @Override
    public void checkClientTrusted(X509Certificate[] chain, String authType, SSLEngine sslEngine) throws CertificateException {
        trustManager.checkClientTrusted(chain, authType, sslEngine);
    }

    @Override
    public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
        trustManager.checkServerTrusted(chain, authType);
    }

    @Override
    public void checkServerTrusted(X509Certificate[] chain, String authType, Socket socket) throws CertificateException {
        trustManager.checkServerTrusted(chain, authType, socket);
    }

    @Override
    public void checkServerTrusted(X509Certificate[] chain, String authType, SSLEngine sslEngine) throws CertificateException {
        trustManager.checkServerTrusted(chain, authType, sslEngine);
    }

    @Override
    public X509Certificate[] getAcceptedIssuers() {
        X509Certificate[] acceptedIssuers = trustManager.getAcceptedIssuers();
        return Arrays.copyOf(acceptedIssuers, acceptedIssuers.length);
    }

    public void setTrustManager(X509ExtendedTrustManager trustManager) {
        this.trustManager = Objects.requireNonNull(trustManager);
    }

}

Usage

// Your key and trust manager created from the pem files
X509ExtendedKeyManager aKeyManager = ...
X509ExtendedTrustManager aTrustManager = ...

// Wrapping it into your hot swappable key and trust manager
HotSwappableX509ExtendedKeyManager swappableKeyManager = new HotSwappableX509ExtendedKeyManager(aKeyManager);
HotSwappableX509ExtendedTrustManager swappableTrustManager = new HotSwappableX509ExtendedTrustManager(aTrustManager);

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(new KeyManager[]{ swappableKeyManager }, new TrustManager[]{ swappableTrustManager })

// Give the sslContext instance to your server or client
// After some time change the KeyManager and TrustManager with the following snippet:

X509ExtendedKeyManager anotherKeyManager = ... // Created from the new pem files
X509ExtendedTrustManager anotherTrustManager = ... // Created from the new pem files

// Set your new key and trust manager into your swappable managers
swappableKeyManager.setKeyManager(anotherKeyManager)
swappableTrustManager.setTrustManager(anotherTrustManager)

So even when your SSLContext instance is cached in your server of client you can still swap in and out new keymanager and trustmanager.

The code snippets are available here:

Github - SSLContext Kickstart

Option 2

If you don't want to add the custom (HotSwappableKeyManager and HotSwappableTrustManager) code to your code base you can also use my library:

<dependency>
    <groupId>io.github.hakky54</groupId>
    <artifactId>sslcontext-kickstart</artifactId>
    <version>7.4.5</version>
</dependency>

Usage

SSLFactory baseSslFactory = SSLFactory.builder()
          .withDummyIdentityMaterial()
          .withDummyTrustMaterial()
          .withSwappableIdentityMaterial()
          .withSwappableTrustMaterial()
          .build();

SSLContext sslContext = sslFactory.getSslContext();
          
Runnable sslUpdater = () -> {
    SSLFactory updatedSslFactory = SSLFactory.builder()
          .withIdentityMaterial(Paths.get("/path/to/your/identity.jks"), "password".toCharArray())
          .withTrustMaterial(Paths.get("/path/to/your/truststore.jks"), "password".toCharArray())
          .build();
    
    SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory);
};

// initial update of ssl material to replace the dummies
sslUpdater.run();
   
// update ssl material every hour    
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.HOURS);

Update #1 - example with pem files In the comments someone requested an example with pem files, so below is an example of refreshing the ssl configuration with pem files:

First make sure you have the following library:

<dependency>
    <groupId>io.github.hakky54</groupId>
    <artifactId>sslcontext-kickstart-for-pem</artifactId>
    <version>7.4.5</version>
</dependency>

And the code example:

SSLFactory baseSslFactory = SSLFactory.builder()
          .withDummyIdentityMaterial()
          .withDummyTrustMaterial()
          .withSwappableIdentityMaterial()
          .withSwappableTrustMaterial()
          .build();

SSLContext sslContext = sslFactory.getSslContext();
          
Runnable sslUpdater = () -> {
    X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial(Paths.get("/path/to/your/certificate-chain.pem"), Paths.get("/path/to/your/"private-key.pem"));
    X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial(Paths.get("/path/to/your/"some-trusted-certificate.pem"));

    SSLFactory updatedSslFactory = SSLFactory.builder()
          .withIdentityMaterial(keyManager)
          .withTrustMaterial(trustManager)
          .build();
    
    SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory);
};

// initial update of ssl material to replace the dummies
sslUpdater.run();
   
// update ssl material every hour    
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.HOURS);
Hakan54
  • 3,121
  • 1
  • 23
  • 37
  • Thanks for the detailed example, I edited the question and added a code snippet for you to see exactly what we see in Spring, namely nothing :). We never see Keystores, SSLContexts or anything, Geode is based on property setting as many other things out there. Normally everything is set through the ClientCacheFactory(), which provides you with the basic object... I am now looking into this one, if we could maybe recreate it with Spring, or maybe just to use Geode client directly and drop Spring but that very short code I provided is so tempting :). – Zoltan Kutasi Jan 25 '21 at 20:23
  • Ah thats sad to hear. I also jumped into their source code to understand how the they have implemented ssl. I only discovered property based configuration, so I don't think it is possible what you are trying to achieve. You can open an issue to ask if they could add this feature on their github – Hakan54 Jan 25 '21 at 23:22
  • Finally I think I figured out, thanks to you and @jens-d. The solution was indeed to set the SSLContext and I used your technique to wrap around my KeyManager. The other trick I had to do is use the default SSLContext as Geode devs support my use-case through that (I will explain in the other comment that part). Now the only thing I missed is that it is forbidden it seems to set the default SSLContext multiple times, It did not result in any error, but only the first setting was done properly and I debugged this part for 10 hours more :). Thanks once again, I accepted your solution. – Zoltan Kutasi Jan 26 '21 at 19:25
  • @Hakan54 Thanks for the solution, but I'm trying to understand how this actually helps refresh the certificate on the fly. The KeyManager relies on the PEM file in order to be created, so then where does the PEM file get refreshed exactly? – ethereumbrella Aug 15 '22 at 21:08
  • @ethereumbrella I updated my answer with example code snippet while using pem files. I hope that answers your question. It is up to you how and when to replace/update the pem file on your file system. You can check whether the file has been changed between two time slots etc – Hakan54 Aug 15 '22 at 21:28
-1

I think what you're looking for is very similar to what the Java buildpack does when deploying apps to CloudFoundry. When an app is deployed, the buildpack injects a custom Security Provider which watches various key/trust stores for changes. This allows the certificates to be updated without needing to restart the app (https://docs.cloudfoundry.org/buildpacks/java/).

I'm not sure of the exact implementation details but the code for the Security Provider can be found here https://github.com/cloudfoundry/java-buildpack-security-provider. Hopefully this will give you some ideas on how to implement this for your own needs.

Jens D
  • 4,229
  • 3
  • 16
  • 19
  • Thanks for the hint. Unfortunately the real issue here is that Geode does not expose it seems the KeyManager/Keystore as I have found browsing the code, and the SSLContext or some wrapper object used for connections is even cached in a few places so it seems impossible to get rid of them. I have tried many things yesterday but found no exposed method I could use to even access these. We already have a file-watcher in our App, we can check if the certificates change, we can act on this event, just I am out of clue what we shall do in this case. – Zoltan Kutasi Jan 25 '21 at 07:46
  • Ah! Sorry I completely forgot to mention the `ssl-use-default-context` Geode property which needs to be set to `true` in order to enable this. https://geode.apache.org/docs/guide/113/managing/security/implementing_ssl.html – Jens D Jan 25 '21 at 13:43
  • Thanks a lot for your helpful comment on that part, I have found this: [GEODE-5338] https://github.com/apache/geode/pull/2244, this is exacly my use case... and the devs say in this ticket that I should use the default SSLContext, so I used Hakan54's solution to wrap my KeyManager into the default SSLContext and refresh that one, as from the code, Geode at each connection fetches the SSLContext if it is set to be the default one, and creates a new one internally if not. So the two solutions together finally seems to have cracked the problem. Thanks again... I accepted the other comment though. – Zoltan Kutasi Jan 26 '21 at 19:28
  • Thanks for the update. I'm glad you were able to get it working. – Jens D Jan 28 '21 at 04:41