Receiving SSLHandshakeException: handshake_failure despite my client ignoring all certs

Question

I have a Java program that connects to a webserver using SSL/TLS, and sends various HTTP requests over that connection. The server is localhost and is using a self-signed cert, but my code is using custom TrustManagers, and ignores invalid certificates. It has worked perfectly until now.

The only difference on the server is that it used to run jboss 6 and is now running jboss 7. I'm not sure if this is a configuration issue, or whether there is a problem with my code, but I get the same errors if I try to connect using other Java-based programs like WebScarab or ZAP.

In any case, is there anything I can do to my code to get around this problem? Here is the error in full:

Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(Unknown Source)

Here are the debug messages before the failure:

main, WRITE: TLSv1 Handshake, length = 75
main, WRITE: SSLv2 client hello message, length = 101
main, READ: TLSv1 Alert, length = 2
main, RECV TLSv1 ALERT: fatal, handshake_failure

score 28 · Accepted Answer · answered Mar 23 '12 at 10:37

28

So I found the problem. There might be a bug in Java, but the client seems to initiate a TLSv1 Handshake, but then sends an SSLv2 client hello message, at which point the server rejects the connection.

This happens even if you create your SSLContext with an instance of TLS:

SSLContext sslContext = SSLContext.getInstance("TLS");

The solution is to set a system property before any connection attempts are made:

System.setProperty("https.protocols", "TLSv1");

There are probably other solutions to it, but this one worked for me.

answered Mar 23 '12 at 10:37

Rsaesha

1,104
1
13
25

@Bruno Yeah, I saw that earlier whilst searching Google. Is this a bug in Java? I can't see how it could/should be default behaviour when I've specified the SSLContext as TLS. – Rsaesha Mar 23 '12 at 14:26
1

It's not a bug, it's by design (the SSLv2 is for backward comp.). See [JSSE Ref/SSLContext](http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#SSLContext): "*For example, getInstance("SSLv3") may return a instance which implements "SSLv3" and "TLSv1"* [...] You can control which protocols are actually enabled for an SSL connection by using the method setEnabledProtocols(String[] protocols)". In your case, `https.protocols` takes care of `setEnabledProtocols` for `HttpsURLConnection`. – Bruno Mar 23 '12 at 14:34
Right, but that doesn't explain why it tries to use SSLv2 client hello messages when I've asked it to use TLS. The server only supported TLSv1 (specifically), and from the debug messages, it looks like Java initiated a TLSv1 handshake, before switching back to SSL (I've updated the original post with the debug messages). – Rsaesha Mar 26 '12 at 08:36
A similar bug haunted me for days. Rsaesha's solution worked like a charm. I tried to connect to a web service exposed via SimpleHttpServerJaxWsServiceExporter. Connecting to it from Java 7 worked without problems. Connecting to it from Java 6 caused the exception. – Philipp Sch May 17 '13 at 09:30

score 4 · Answer 2 · answered Jan 29 '15 at 19:48

Was this ever resolved?

I had the exact same problem, essentially I was receiving a handshake exception immediately following the clientHello. So The chain of events was

I would present my certificate to the server
Server would imediately respond with a handshake failure. (I would not even get a Server Hello back).

Eventually I found that the server was requiring a stronger encryption/decryption algorithm than what I Was supplying in the initial handshake phase (Ie. Client and Server could not agree on a mutual encryption algorithm to use for the ssl communication).

I need to install the Unlimited Java JCE (Java Cryptography Extension Policy). There are export rules on using this, so if you ship your code overseas that may have implications..however this is what solved my problem.

This link explains how to install the updated policies http://suhothayan.blogspot.com/2012/05/how-to-install-java-cryptography.html

This was also a great link that helped me understand exactly what was going on https://support.f5.com/kb/en-us/solutions/public/15000/200/sol15292.html#id

This may or may not be the issue, but when the handshake fails immediately after the client Hello, it looks like the client and the server can not agree on something (in many cases its the encryption algorithms that they will mutually need to communicate).

score 4 · Answer 3 · answered Mar 22 '12 at 22:14

4

The info you provide is very little as well as your stack trace.
I'll take a guess here.
What I suspect is that in the new server the protocol is TLSv1 while your clients try to connect with SSLv3 (or less) and as a result the handshake fails.

Change you clients to use higher version of TLS or
Make your webserver support SSLv3 as well. I know how to do this in Tomcat but not in JBoss.

If this doesn't work update the post with more info (and a full stack trace).
You should enable ssl debug info -Djavax.net.debug=ssl

answered Mar 22 '12 at 22:14

Cratylus

52,998
69
209
339

The full stack trace just shows the connection method of my class, and whatever methods call that. It is the same method that has been working with previous versions of the server. Anyway, here is the debug: http://pastebin.com/RJqcQwaP What seems a bit odd to me is that even though I tell my program to use TLS, it seems to want to send out SSLv2 client hello messages. – Rsaesha Mar 23 '12 at 10:15
For me also facing similar problem details here - https://serverfault.com/questions/954489/ssl-routinesssl3-read-bytestlsv1-alert-decrypt-error-on-mutual-authendication How can we Identify this decrypt error ? – Shankar Feb 20 '19 at 11:22

uaarkoti · Answer 4 · 2012-03-23T01:05:29.103

You are seeing this error most probably because the keystore that your JBoss 6 had access to is not accessible to your JBoss 7 instance.

What I would recommend is the following.

Your self-signed server certificate must be imported into a truststore

keytool -import -alias gridserver -file server.crt -storepass $YOUR_PASSWORD_HERE -keystore server.keystore

Add the following properties to your run.conf

-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.trustStoreType=jks
-Djavax.net.ssl.keyStore=clientcertificate.p12
-Djavax.net.ssl.trustStore=server.keystore
-Djavax.net.debug=ssl # very verbose debug. Turn this off after everything looks good.
-Djavax.net.ssl.keyStorePassword=$YOUR_PASSWORD_HERE
-Djavax.net.ssl.trustStorePassword=$YOUR_PASSWORD_HERE

The server runs fine through a web browser like Chrome or Firefox, so the keystore on the server isn't the issue. Certification shouldn't be an issue either, since I am simply ignoring all certs. — Rsaesha, Mar 23 '12 at 09:57

score 0 · Answer 5 · answered Jun 12 '14 at 02:47

0

I think this is related to a Java 7 bug. It is hard to be sure without more details.

answered Jun 12 '14 at 02:47

Michael Munsey

3,740
1
25
15

score 0 · Answer 6 · answered Jan 16 '18 at 07:35

0

For me solution was : System.setProperty("https.protocols", "TLSv1.1,TLSv1.2");

answered Jan 16 '18 at 07:35

Mahmoud Saleh

33,303
119
337
498

Bruno Grieder · Answer 7 · 2012-03-23T09:49:03.687

0

The stack trace is from you client code and your client 'Received [a] fatal alert'. In other words, the SSL error happened in Jboss, not your client.

Your client side custom TrustManagers have therefore nothing to do with it. My wild guess is that your new Jboss 7 is configured to require client certificate and your client did not present any.

To debug your SSL connection, use openssl and try this:

openssl s_client -connect jboss.server.com:443

or is it is an SSLV3 server

openssl s_client -connect jboss.server.com:443 -ssl3

This should print a lot of interesting information.

edited Mar 23 '12 at 09:49

answered Mar 23 '12 at 09:33

Bruno Grieder

28,128
8
69
101

My wild guess was just that, wild. Your server is fine. user384706 got you on the right track. This is more likely a protocol version mismatch. I see your server is TLSv1/SSLv3 and you seem to be sending SSLv2. – Bruno Grieder Mar 23 '12 at 10:34

Receiving SSLHandshakeException: handshake_failure despite my client ignoring all certs

7 Answers7

Linked

Related