If there's more than one certificate in Jetty's key store, how does it choose?

Question

There is some code in our system for automatically generating self-signed certificates into a key store which is then used by Jetty. If a key for a given host already exists then nothing happens but if it doesn't exist, we generate a new key, like this:

public void generateKey(String commonName) {
    X500Name x500Name = new X500Name("CN=" + commonName);
    CertAndKeyGen keyPair = new CertAndKeyGen("DSA", "SHA1withDSA");
    keyPair.generate(1024);
    PrivateKey privateKey = keyPair.getPrivateKey();
    X509Certificate certificate = keyPair.getSelfCertificate(x500Name, 20*365*24*60*60);
    Certificate[] chain = { certificate };
    keyStore.setEntry(commonName, privateKey, "secret".toCharArray(), chain);
}

This all works fine as long as there is only one key and certificate in the key store. Once you have multiple keys, weird things happen when you try to connect:

java.io.IOException: HTTPS hostname wrong:  should be <127.0.0.1>

This was quite a mystifying error but I finally managed to track it down by writing a unit test which connects to the server and asserts that the CN on the certificate matches the hostname. What I found was quite interesting - Jetty seems to arbitrarily choose which certificate to present to the client, but in a consistent fashion.

For instance:

• If "CN=localhost" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
• If "CN=127.0.0.1" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
• If "CN=192.168.222.100" (cheese.mydomain) and "CN=cheese.mydomain" are in the key store, it always chose "CN=192.168.222.100".

I wrote some code which loops through the certificates in the store to print them out and found that it isn't consistently choosing the first certificate or anything trivial like that.

So exactly what criteria does it use? Initially I thought that localhost was special but then the third example baffled me completely.

I take it that this is somehow decided by the KeyManagerFactory, which is SunX509 in my case.

Answer 1

This is indeed ultimately decided by the KeyManager (generally obtained from a KeyManagerFactory).

A keystore can have a number of certificates stored under different aliases. If no alias is explicitly configured via certAlias in the Jetty configuration, the SunX509 implementation will pick the first aliases it finds for which there is a private key and a key of the right type for the chosen cipher suite (typically RSA, but probably DSA in your case here). There's a bit more to it to the choice logic, if you look at the Sun provider implementation, but you shouldn't really rely on the order in general, just the alias name.

You can of course give Jetty your own SSLContext with your own X509KeyManager to choose the alias. You would have to implement:

 chooseServerAlias(String keyType, Principal[] issuers, Socket socket)

Unfortunately, apart from keyType and issuers, all you get to make the decision is the socket itself. At best, the useful information you get there are the local IP address and the remote one.

Unless your server is listening to multiple IP addresses on the same port, you will always get the same local IP address. (Here, obviously, you have at least two: 127.0.0.1 and 192.168.222.100, but I suspect you're not really interested in localhost except for your own tests.) You would need Server Name Indication (SNI) support on the server side to be able to make a decision based on the requested host names (by clients that support it). Unfortunately, SNI was only introduced in Java 7, but only on the client side.

Another problem you will face here is that Java clients will complain about IP addresses in the Subject DN's CN. Some browsers would tolerate this, but this is not compliant with the HTTPS specification (RFC 2818). IP addresses must be Subject Alternative Name entries of IP-address type.