SSLHandshakeException: Received fatal alert: handshake_failure after Java 6 -> 8 upgrade

Question

We've recently updated a project from Java 6 to Java 8 and now we've hit a brick wall regarding SSL handshake.

The service layer uses a client to request and receive calls from a third party application. In the service layer, the keystore is initialized with

    System.setProperty("javax.net.ssl.trustStore", keyStoreFile);
    System.setProperty("javax.net.ssl.trustStorePassword", keyStorePassword);

and injected via applicationContext.xml:

    <property name="keyStoreFile" value="/keystore/keystore.keystore" />
    <property name="keyStorePassword" value="password" />

The client is supposed to trust all certificates in case of errors:

private void trustHttpsCertificates() throws Exception {

    try {
        Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());

        // Create a trust manager that does not validate certificate chains 
        TrustManager[] trustAllCerts = new TrustManager[] { 
            new X509TrustManager() {
                public X509Certificate[] getAcceptedIssuers() {
                    return new X509Certificate[0];
                }

                public void checkServerTrusted(X509Certificate[] certs, String authType) {
                }

                public void checkClientTrusted(X509Certificate[] certs, String authType) {
                }
            }
        };

        // Ignore differences between given hostname and certificate hostname 
        HostnameVerifier hv = new HostnameVerifier() {
          public boolean verify(String hostname, SSLSession session) { return true; }
        }; 

        // Install the all-trusting trust manager 
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCerts, new SecureRandom());
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        HttpsURLConnection.setDefaultHostnameVerifier(hv);
    } catch (KeyManagementException e) {
        String errorMsg = "client initialization error: " + e.getMessage();
        log.error(errorMsg);
        throw new Exception(errorMsg, e);
    } 
}

I changed "SSL" above to "TLSv1" in the upgrade process, since SSL isn't supported on Java 8. Might that be a problem?

Debug logs

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
http-nio-9080-exec-6, setSoTimeout(0) called
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1
Ignoring unsupported cipher suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 for TLSv1.1
Ignoring unsupported cipher suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 for TLSv1.1
%% No cached client session
*** ClientHello, TLSv1.2
RandomCookie:  GMT: 1423711122 bytes = { 237, 188, 53, 112, 79, 112, 248, 92, 164, 127, 178, 34, 205, 40, 245, 25, 77, 143, 116, 126, 203, 96, 61, 181, 114, 148, 66, 227 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA, MD5withRSA
***
[write] MD5 and SHA1 hashes:  len = 237
// ...
http-nio-9080-exec-6, READ: TLSv1 Alert, length = 2
http-nio-9080-exec-6, RECV TLSv1.2 ALERT:  fatal, handshake_failure
http-nio-9080-exec-6, called closeSocket()
http-nio-9080-exec-6, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
// ...
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1512)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)

Thoughts

There are mainly two things that stand out compared to the old version with Java 6:

In the logs for the old verison, when a successful attempt is made, it clealy presents the correct certificate:

http-9080-1, READ: TLSv1 Handshake, length = 1375
*** Certificate chain
chain [0] = [
[
  Version: V3
  Subject: xxx
  Signature Algorithm: MD5withRSA

  Key:  Sun RSA public key, 1024 bits
  Validity: [From: Wed May 26 14:31:31 CEST 1999,
               To: Thu May 25 14:31:31 CEST 2000]
  Issuer: xxx
  SerialNumber: [    01]
]

Algorithm: [MD5withRSA]

This doesn't happen in the new Java 8 version.

Also, TLSv1 vs TLSv1.2?

http-nio-9080-exec-6, READ: TLSv1 Alert, length = 2
http-nio-9080-exec-6, RECV TLSv1.2 ALERT:  fatal, handshake_failure

Is it saying I'm trying to connect with TLSv1.2? Or TLSv1? And it's not accepted? I don't really understand. Is there a way to find out what TLS versions are accepted by the server?

I've tried adding flags to startup:

-Dhttps.protocols=TLSv1
-Ddeployment.security.TLSv1=true
-Djavax.net.ssl.keyStore=C:\keystore\keystore.keystore
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStore=C:\keystore\keystore.keystore
-Djavax.net.ssl.trustStorePassword=password

Also adding a keystore Manager programmatically:

KeyStore ks = KeyStore.getInstance("JKS");
InputStream ksIs = new FileInputStream("/keystore/keystore.keystore");

try {
    ks.load(ksIs, password.toCharArray());
} catch (Exception e) {
    e.printStackTrace();
} finally {
    if (ksIs != null) {
        try {
            ksIs.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(ks, password.toCharArray());

and then initializing SSLContext with that:

sc.init(kmf.getKeyManagers(), trustAllCerts, new SecureRandom());

but the problem persists. Help, anyone?

This http://stackoverflow.com/a/34296145/185565 solution works in JDK7 and JDK8. Use a facade SSL factory then custom hostnameverifier can be used for SSL connections. You may wrap `sc.getSocketFactory()` instance and should work. — Whome, Jan 18 '17 at 23:18

Jesper N · Accepted Answer · 2020-06-11T19:32:28.493

Alright, so we've got it working now. I'll post the answer here in case someone might need it some day.

We have tried quite a few things, so I'm not exactly sure what actually needed to be done in order for it to work, but here are some of the things we changed in the process.

 -Dhttps.protocols=SSLv3,TLSv1,SSLv2Hello

Adding this flag led to the certificate being presented in the javax.net.debug logs, but we were still getting SSLHandshakeException. It seems like the only cipher the server would accept was SSL_RSA_WITH_RC4_128_MD5. This was not picked automatically by our client.

-Dhttps.cipherSuites=SSL_RSA_WITH_RC4_128_MD5

We added this flag to restrict the cipher suits for the client. Together with setting the same restriction programmatically (not sure if both are needed):

socket.setEnabledCipherSuites(new String[] {"SSL_RSA_WITH_RC4_128_MD5"});

Restricting the available cipher suites to the only one that the client could use, made the client pick that cipher suite.

We also did the following changes the the jre/lib/security/java.security file to enable SSLv3 and the SSL_RSA_WITH_RC4_128_MD5 cipher:

remove SSLv3 from jdk.tls.disabledAlgorithms
add SSL_RSA_WITH_RC4_128_MD5 to jdk.tls.legacyAlgorithms

This is probably not recommended for production servers, since SSLv3 is obsolete, and the cipher is very old and outdated, but in this case security is not a huge concern (internal application use).

These posts were also helpful to me:

Interesting, I had a similar problem Java7 fine but Java8 failed. I use `HttpsURLConnection.setSSLSocketFactory(..), HttpsURLConnection.setHostnameVerifier(new DebugAllowAllVerifier())` code. Removing a verifier fixed a problem. I could not find out why a simple `return true` verifier is a problem. — Whome, Jan 18 '17 at 20:04
This http://stackoverflow.com/a/34296145/185565 solution works using a facade SSLFactory and setHostnameVerifier() together. Strange because it does nothing but is still ok on JDK7+JDK8. — Whome, Jan 18 '17 at 23:20

Steffen Ullrich · Answer 2 · 2015-08-25T16:05:41.770

2

My guess is that your server is too broken to deal properly with TLS 1.2 handshakes. Usually a server which does not understand TLS 1.2 should reply with the best version it can but your server does not.

Broken servers like this exists and browsers try to work around these by retrying with a lower TLS version. Outside of browsers these retries are not common so these clients simply fail.

While I cannot say for sure that the server is broken the certificate which expired 15 years ago and was signed with the long broken MD5 algorithm suggest that you have to do with a very old and neglected installation. So chances are high that it never occurred to the developers of the original server that something like TLS 1.2 might ever exist or that the it croaks on one of the TLS extensions used in the TLS 1.2 handshake.

Since this issue is not related to the validation of the certificate all attempts to fix the issue by fiddling in the area of validation are useless. You might have more success if you enforce the use of TLS 1.1 or TLS 1.1 instead of TLS 1.2. You might try to do this with the -Dhttps.protocols=TLSv1,TLSv1.1 or -Dhttps.protocols=TLSv1 settings.

edited Aug 25 '15 at 16:05

answered Aug 25 '15 at 15:59

Steffen Ullrich

114,247
10
131
172

Thanks for the reply. That's my guess too. Does "ClientHello, TLSv1.2" means my client is trying to use TLSv1.2? As you can see from my original post, I've already tried using -Dhttps.protocols=TLSv1 but it doesn't change anything. Is there any other way, programmatically or with flags, to enforce TLSv1 (or maybe even SSLv3 (I know, it shouldn't be used, but in this case security is not a very big deal))? – Jesper N Aug 25 '15 at 16:26
@JesperLehtinen: Yes, the client does a TLS 1.2 handshake. It might actually be that 'TLSv1' includes the later version, the documentation is not very specific in this regard. With this old and insecure server you might actually try to use 'SSLv3' - I doubt that this can lessen the security of this server even more. – Steffen Ullrich Aug 25 '15 at 17:55
Thank you for your help Steffen, now it's working again. I'll post the solution and followup in a new answer. – Jesper N Aug 26 '15 at 12:10

score -4 · Answer 3 · edited Nov 08 '17 at 13:14

String testURL = "https://api.chargeio.com/status";
SSLContext sslcontext = SSLContext.getInstance("TLSv1.2");
sslcontext.init(null, null, null);
try {
    SSLConnectionSocketFactory socketFactory = new
    SSLConnectionSocketFactory(sslcontext,
        SSLConnectionSocketFactory.BROWSER_COMPATIBLE_HOSTNAME_VERIFIER); // Socket
    HttpClient client =
        HttpClients.custom().setSSLSocketFactory(socketFactory).build();
    HttpGet httpget = new HttpGet(testURL);
    HttpResponse response = client.execute(httpget);
    System.out.println(EntityUtils.toString(response.getEntity()));
    System.out.println("Response Code (Apache): " + response.getStatusLine().getStatusCode());
} catch (Exception e) {
    System.err.println("HttpsURLConnection Failed");
    e.printStackTrace();
}

SSLHandshakeException: Received fatal alert: handshake_failure after Java 6 -> 8 upgrade

3 Answers3

Linked