Java - HTTPS GET call fails from EC2 instance with handshake_failure, works locally

Question

I'm running out of ideas, which is why I'm asking for help here. I have a small class that does a REST call over HTTPS. I'm doing some kind of scraping, and would like to avoid installing all SSL certificates if possible.

The code works well locally (both from Eclipse and a standalone Tomcat), but it fails with a javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure every time when running from my AWS EC2 instance.

Code is:

// apiUrl looks like https://hsreplay.net/api/v1/games/jdUbSjsEcBL5rCT7dgMXRn
private String restGetCall(String apiUrl) throws Exception {
    System.setProperty("javax.net.debug", "ALL");
    SSLContextBuilder builder = new SSLContextBuilder();
    builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
    SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build(),
            SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER) {
        @Override
        protected void prepareSocket(SSLSocket socket) throws IOException {
            try {
                log.debug("************ setting socket HOST property *************");
                PropertyUtils.setProperty(socket, "host", "hsreplay.net");
                socket.setEnabledProtocols(new String[] { "SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2" });
            }
            catch (IllegalAccessException | NoSuchMethodException | InvocationTargetException ex) {
                log.error(ex.getMessage());
                slackNotifier.notifyError(ex);
            }
            super.prepareSocket(socket);
        }

    };
    CloseableHttpClient httpClient = HttpClients.custom().setSSLSocketFactory(sslsf).build();

    HttpGet httpGet = new HttpGet(apiUrl);
    CloseableHttpResponse response1 = httpClient.execute(httpGet);
    StringBuilder result = new StringBuilder();
    try {
        System.out.println(response1.getStatusLine());
        HttpEntity entity1 = response1.getEntity();
        BufferedReader rd = new BufferedReader(new InputStreamReader(entity1.getContent()));
        String line;
        while ((line = rd.readLine()) != null) {
            result.append(line);
        }
        // do something useful with the response body
        // and ensure it is fully consumed
        EntityUtils.consume(entity1);
    }
    finally {
        response1.close();
    }
}

I also tried the basic version (without any override of the SSLFactory methods), but to no avail.

Also, I don't manage to get the logs from javax.net.debug on Tomcat (neither on the remote nor on a local one), so if you have any idea here that could also be very useful.

Please let me know if I can provide more information, and thanks a lot for your help!

Sébastien

Have you seen [this](http://stackoverflow.com/a/6353956/1695906)? Handshake failure has a number of different causes, but with an up-to-date trust store, it seems like you should be able to make this work without installing specific certificates. It appears to be an ECC certificate from CloudFlare, so I'd say it's relatively new technology on the server side, but not outside the mainstream. — Michael - sqlbot, Aug 29 '16 at 04:19
@Michael-sqlbot: a handshake failure is (almost?) never associated with validation of the server certificate and thus changes on the trust store will not help. — Steffen Ullrich, Aug 29 '16 at 05:01
@Michael-sqlbot Yes, I've seen that. I'd come to the same conclusion as Steffen, so I haven't tinkered with the truststore yet, but might be something to test if I'm still stuck :) — Sébastien Tromp, Aug 29 '16 at 07:39
@SteffenUllrich thanks, I was trying to address/dismiss *"would like to avoid installing all SSL certificates if possible"* but didn't communicate my thoughts very well, there. I intended for that sentence to indicate that explicitly trusting ("installing") the server's cert shouldn't be necessary as part of the solution unless the trust store was the (unlikely) root cause, and I see that it didn't really come out that way. I apologize for the lack of clarity. — Michael - sqlbot, Aug 29 '16 at 09:47

score 2 · Answer 1 · answered Aug 29 '16 at 05:17

2

Handshake failures might be caused by lots of different reasons, like the wrong Java version, server settings, setting a hostname verifier (bad idea anyway), SNI ... . The current information in this question are not enough to say for sure what the problem is.

When looking at the SSLLabs report for the server two information look interesting:

The server needs SNI.
The server supports only ECDHE ciphers.

The first might be a problem with older Java versions or even with JDK8 if you set a hostname verifier (which you do although it would not be necessary for this site).

The second might be a problem with older Java version or if the cryptography extensions for unlimited strength are not installed.

answered Aug 29 '16 at 05:17

Steffen Ullrich

114,247
10
131
172

Thanks Steffen. Do you know what additional information might be useful? Java version is openJDK 1.7.0_85, which is the same I use in my local environment where I don't get any errors. So could the java version possibly be the cause? The hostname verifier should be one from Apache's HTTPClient that always returns true. – Sébastien Tromp Aug 29 '16 at 06:44
@SébastienTromp: I recommend to take a look at all the other questions which deal with this issue, i.e. what information they provide. And maybe there is already an answer there which exactly fits your environment. But typically enabling TLS debugging is a good choice or a packet capture so that one can see the details of the TLS handshake. – Steffen Ullrich Aug 29 '16 at 06:47
I had a look at basically everything I could find :) I'll look more into TLS debugging - currently it looks like Tomcat doesn't capture the output, but it's probably my best chance. Thanks! – Sébastien Tromp Aug 29 '16 at 06:59
Printing the enabled ciphers returned `TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_EMPTY_RENEGOTIATION_INFO_SCSV`, so no ECDHE cipher. They are apparently not supported by OpenJDK 7, and http://stackoverflow.com/questions/31971499/ecdhe-cipher-suites-not-supported-on-openjdk-8-installed-on-ec2-linux-machine suggests a way to support them – Sébastien Tromp Aug 29 '16 at 07:50
Indeed, your suggestion put me on the correct rails. The openJDK 7 doesn't support ECDHE ciphers. I had the options to add the appropriate cryptography libraries, or migrate to Sun's JDK. I did the latter, and it works now. Thanks again! – Sébastien Tromp Aug 29 '16 at 12:42

score 0 · Answer 2 · answered Aug 29 '16 at 12:44

0

@SteffenUlrich's answer put me on the correct rails. The issue was indeed because of the ECDHE ciphers that aren't supported in open jdk 7. I migrated to Oracle's JDK 8, and things are working fine now.

answered Aug 29 '16 at 12:44

Sébastien Tromp

603
1
15
30

Java - HTTPS GET call fails from EC2 instance with handshake_failure, works locally

2 Answers2