Using SSLEngine to use a certificate signed for an internal network location

Question

Background

I'm trying to create 2 java applications that talk to each other over SSL on an internal network. I'm doing this by giving an SSLEngine access to a keystore that contains a private key entry for the certificate to be used.

   private static SSLEngine createSslEngine()
   {
      try
      {
         KeyStore ks = KeyStore.getInstance( KeyStore.getDefaultType() );
         char[] pwdArray = "changeit".toCharArray();
         ks.load( null, pwdArray );
         try ( InputStream ksIs = ServerAssociationConnector.class.getResourceAsStream( "clientkeystore" ) )
         {
            ks.load( ksIs, pwdArray  );
         }
         KeyManagerFactory kmf = KeyManagerFactory.getInstance( KeyManagerFactory.getDefaultAlgorithm() );

         kmf.init( ks, pwdArray );
         SSLContext context = SSLContext.getInstance( "TLSv1.3" );
         context.init( kmf.getKeyManagers(), null, null );

         return context.createSSLEngine();
      }
      catch ( NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException | CertificateException | IOException | KeyManagementException exception )
      {
         throw new AssertionError( "SSLException", exception );
      }
   }

This works fine when using root certificates (hitting dubrovnik:2762 in a browser complains about the untrusted certificate but otherwise works). However; I would like to use a non root certificate that is signed by an internal CA just for the domain in question (called dubrovnik in this example).

When I instead sign a certificate with the a root certificate and have dubrovnik in the subject alternative name java refuses to use the certificate and instead produces a javax.net.ssl.SSLHandshakeException: No available authentication scheme exception

The keystore I'm using contains this:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

dubrovnik, 2 Nov 2020, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 88:5F:53:7C:85:8E:65:01:3E:E1:E8:F0:D6:17:7A:8B:22:EF:11:DD:5F:E6:30:FE:A7:3B:F1:FA:07:C8:46:38
thecaroot, 2 Nov 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): 9E:2F:86:B6:17:83:D2:26:88:42:49:E4:3F:DA:DA:19:31:11:18:F7:15:6D:16:35:C4:3E:1B:E4:F8:E6:FC:3A

How I've been creating my certificates

I've been using openssl and keytool to create the local ca and signed certificate

openssl genrsa -des3 -out myCA.key 2048
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem

##create the certificate to sign
keytool -keystore clientkeystore -genkey -alias dubrovnik -dname "CN=dubrovnik" -ext san=dns:dubrovnik
##create a request to have certificate signed
keytool –certreq –keystore clientkeystore –alias dubrovnik –keyalg rsa –file dubrovnik.csr

##CREATE A FILE CALLED dubrovnik.conf as described in https://stackoverflow.com/a/47779814/2187042

##sign the request
openssl x509 -req -CA myCA.pem -CAkey myCA.key -in dubrovnik.csr -out dubrovnik.cer -days 3650 -CAcreateserial -extfile dubrovnik.conf -extensions v3_req
##import the root level certificate (just the certificate, not the private key)
keytool -import -keystore clientkeystore -file myCA.pem -alias theCARoot 
##import the signed client certificate
keytool -import -keystore clientkeystore -file dubrovnik.cer -alias dubrovnik

Question

What do I need to do to make an internal network certificate (not root certificate) acceptable to SSLEngine and avoid a javax.net.ssl.SSLHandshakeException: No available authentication scheme exception

The "javax.net.ssl.SSLHandshakeException: No available authentication scheme" results from a missing "confirmation" - you are using self created certificates that are unknown to the others party. You need to use a **second** keystore but in function of a **truststore** (on both sides) to establish the SSL connection. The SSLEngine checks the certificate against the truststore and gets the "confirmation" that the certificate is valid. — Michael Fehr, Nov 02 '20 at 12:40
I think my problem may have been the same as https://stackoverflow.com/a/61641857/2187042. I was generating DSA certificates (and java was ignoring them) rather than the RSA certificates I should have been using — Richard Tingle, Nov 02 '20 at 13:55
@MichaelFehr: incorrect/missing truststore will produce a different exception and not EVER the exception in this Q. This is like telling someone whose (traditional gasoline-powered) car won't run because the spark plugs have been removed that they need to put more fuel in the tank, but fortunately OP ignored it. — dave_thompson_085, Nov 02 '20 at 18:43

score 2 · Accepted Answer · answered Nov 02 '20 at 15:50

I believe my problem was actually just that the root certificate was an RSA certificate but the non-root certificate was DSA, and DSA is deprecated (and so was being ignored).

This problem was also encountered at javax.net.ssl.SSLHandshakeException: No available authentication scheme

The DSA nature of the certificate can be seen in the certificate details

The keygen command needs to be altered to use RSA

keytool -keystore clientkeystore -genkey -alias dubrovnik –keyalg rsa

Using SSLEngine to use a certificate signed for an internal network location

Background

How I've been creating my certificates

Question

1 Answers1