Is there any way to specify the range for the Diffie-Hellman prime using javax.net.ssl.SSLServerSocket?

Question

I'm trying to reproduce the issue that was raised in Java: Why does SSL handshake give 'Could not generate DH keypair' exception?.

I have a log that shows an exception with this cause:

Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Could not generate DH keypair
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1844)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1827)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1753)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:127)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.apache.axis2.transport.http.AxisRequestEntity.writeRequest(AxisRequestEntity.java:94)
    ... 27 more
Caused by: java.lang.RuntimeException: Could not generate DH keypair
    at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:136)
    at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:621)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:205)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    ... 30 more
Caused by: java.security.InvalidAlgorithmParameterException: Prime size must be multiple of 64, and can only range from 512 to 1024 (inclusive)
    at com.sun.crypto.provider.DHKeyPairGenerator.initialize(DHKeyPairGenerator.java:120)
    at java.security.KeyPairGenerator$Delegate.initialize(KeyPairGenerator.java:658)
    at sun.security.ssl.DHCrypt.<init>(DHCrypt.java:127)
    ... 38 more

I'm using Axis2 to call a web service. I also have the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" in jre/lib/security (but as someone mentioned on the other post this didn't work for them).

For reproducing this issue I'm trying to host a web service in a separate deployment with a similar environment (also using Axis2). This has a custom SSLServerSocketFactory which is creating and configuring the SSLServerSocket.

Using Wireshark (and also javax.net.debug JVM argument) I can see that the cipher suite that is negotiated is TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA but the Prime DH Server Param in the ServerKeyExchange is only 96 bytes (768 bits).

Is there any way I can configure the SSLServerSocket to use a larger prime during the handshake?

I'm guessing the key is being generated by the security provider for RSA from the JRE. Do I need to add my own provider and add it to jre/lib/security/java.security?

Appologies if this is a silly question, I'm new to DHE an Cipher Suites in general. I just need to know enough to get the server test to work (I'm not touching the client implementation).

The duplicate says it was fixed four years ago. The solution is therefore to upgrade your Java version. — user207421, Apr 07 '15 at 05:40
My question is quite different. I am trying to implement a server that uses a prime larger than 1024 bits for DHE RSA key generation. — steinybot, Apr 07 '15 at 05:55
I am trying to generate that situation not solve it. As I stated I'm new to this sort of thing so if it is the provider (not the policy) that is causing the same restriction for client and server then please add that as an answer. Even if I were to use a different provider is this automatically going to be larger than 1024 bits or do I need to configure it? — steinybot, Apr 07 '15 at 06:08
It is the Java version that is providing the limitation, as it says in the duplicated question. If you want to reproduce it, use the appropriate version. — user207421, Apr 07 '15 at 10:27
What I'm confused about is that it also says that you can use BouncyCastle's JCE implementation indicating that perhaps it is the security provider. — steinybot, Apr 08 '15 at 00:36

score 0 · Answer 1 · answered Apr 08 '15 at 00:59

So with the push in the right direction from EJP I eventually managed to track down exactly where the strength (and subsequently the DH parameters) is determined. It is indeed in the JRE (specifically in the ServerHandshaker).

Using the security providers with an older JRE with the strength limitation (as mentioned in the original post) will not work here. While using other security providers will probably prevent this error client side the strength is already determined before the implementation of the KeyPairGenerator is initialized.

As a side note I didn't bother upgrading the JRE to reproduce this. I just added the BouncyCastle JCE provider with debug info, put a breakpoint in the org.bouncycastle.jcajce.provider.asymmetric.dh.KeyPairGeneratorSpi.initialize(int, SecureRandom) method and manually changed the strength. This was good enough for my tests.

Is there any way to specify the range for the Diffie-Hellman prime using javax.net.ssl.SSLServerSocket?

1 Answers1