Apache http client defaults don't work with sni

Question

PRIMARY QUESTION

Is there a bug I'm not aware of with the latest 4.x apache http client? or a setting that I'm missing to get sni to work?

On/in the same jvm (in this case 1.7.0_45).

I run the following two pieces of code to pull an https url that requires SNI for it to connect correctly. The 1st piece of code uses java's built in java.net.* approach, and the 2nd piece of code uses org.apache.http.* (version 4.5) approach.

Approach with java.net.*

public final class JavaNetHitApi extends BaseHitApi {

    @Override
    public JSONObject hitAPI(final String url) throws IOException {
        URL toUse = new URL(url);
        URLConnection conn = toUse.openConnection();
        final String result = readToString(conn.getInputStream());
        JSONObject toReturn = new JSONObject(result);
        return toReturn;
    }

}

Approach with org.apache.http.*

public final class ApacheHttpHitApi extends BaseHitApi {

    @Override
    public JSONObject hitAPI(final String url) throws IOException {
        final HttpGet httpGet = new HttpGet(url);
        try (final CloseableHttpClient httpclient = HttpClients.createDefault();
                final CloseableHttpResponse response = httpclient.execute(httpGet)) {
            String result = readToString(response.getEntity().getContent());
            JSONObject toReturn = new JSONObject(result);
            return toReturn;
        }
    }
}

For the 1st case (java.net.*), it works fine, and returns with no problems.

For the 2nd case (org.apache.http.*), it throws the following error:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Keep in mind this is running on the same JVM, with the same sni enabled settings, etc. The only explanations I've come across through searching all talk about going from java 1.6->1.7, or 1.7->1.8, or enabling sni on the system, etc. In this case, all those variables are eliminated, since the underlying system settings are the same for both pieces of code.

PRIMARY QUESTION RECAP

Is there a bug I'm not aware of with the latest 4.x apache http client? or a setting that I'm missing to get sni to work?

When running with the

"-Djavax.net.debug=all"

flag, I can see where the two approaches differ, namely that the java.net.* approach spits out the line:

Extension server_name, server_name: [host_name: obfuscatedsubdomain.execute-api.us-west-2.amazonaws.com]

And the apache http client does not.

The output snippets are included below for reference.

java.net.* output

Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
main, setSoTimeout(0) called
%% No cached client session
*** ClientHello, TLSv1
RandomCookie:  GMT: 1424292809 bytes = { 117, 134, 25, 51, 251, 228, 170, 18, 240, 70, 32, 245, 131, 90, 128, 105, 26, 220, 234, 92, 81, 47, 129, 195, 17, 137, 210, 125 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension server_name, server_name: [host_name: obfuscatedsubdomain.execute-api.us-west-2.amazonaws.com]
***

apache.org.http.* output

main, setSoTimeout(0) called
Allow unsafe renegotiation: false
Allow legacy hello messages: true
Is initial handshake: true
Is secure renegotiation: false
%% No cached client session
*** ClientHello, TLSv1
RandomCookie:  GMT: 1424293231 bytes = { 222, 180, 202, 14, 219, 36, 94, 157, 23, 55, 5, 98, 165, 33, 227, 160, 201, 191, 122, 194, 241, 154, 216, 137, 159, 81, 208, 143 }
Session ID:  {}
Cipher Suites: [TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods:  { 0 }
Extension elliptic_curves, curve names: {secp256r1, sect163k1, sect163r2, secp192r1, secp224r1, sect233k1, sect233r1, sect283k1, sect283r1, secp384r1, sect409k1, sect409r1, secp521r1, sect571k1, sect571r1, secp160k1, secp160r1, secp160r2, sect163r1, secp192k1, sect193r1, sect193r2, secp224k1, sect239k1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
***

If it supports SNI, it should be enabled by default, and I'd be surprised if there's even a way to disable it in config. — ZhongYu, Sep 01 '15 at 22:51
that was my thought as well, which is why this behaviour seemed odd/buggy. — CasualT, Sep 01 '15 at 22:56
HttpClient merely uses functionality provided by JSSE. HttpClient is known to fully support SNI since 4.3.2 when running on JRE 1.7 — ok2c, Sep 02 '15 at 09:53
feel free to post an sni-enabled url for me to test against with my code/prove me wrong. — CasualT, Sep 02 '15 at 16:27
ok. I've spend the last few hours trying out on different systems/settings. I think I've been able to narrow it down to some other dependency or library causing the issue (since when I run a stripped down test project with only the http library and json dependencies, it works fine). I'll report back once I know more. — CasualT, Sep 02 '15 at 20:15
I don't usually expect a maven dependency to secretly include dependencies that are unlisted in its pom, but I guess if that is normal for you... — CasualT, Sep 08 '15 at 16:44

score 3 · Accepted Answer · answered Sep 02 '15 at 20:57

The issue ended up being that another dependency was not only using a different version of the httpclient, BUT was also packaging it into its jar without listing it as a transitive dependency. (meaning, that it doesn't show up when you go to inspect the pom dependency hierarchy, so it can't be flagged out as the "used version").

The dependency in question was:

<dependency>
    <groupId>com.google.gwt</groupId>
    <artifactId>gwt-dev</artifactId>
    <version>2.7.0</version>
</dependency>

It has the code for apache http included in its jar, and unlisted in its pom. And the version it references is 4.3.1, which exhibits this problem. (4.3.2 and onward do not).

There are two workarounds:

Include the http client dependency 1st in the list within the pom.xml
Remove the gwt-dev dependency from the pom.xml

Ouch, that's really sneaky. Personally I'd recommend always using the [duplicate-finder-maven-plugin](https://github.com/basepom/duplicate-finder-maven-plugin/wiki). This plugin will detect duplicate classes in the classpath (which is nearly always a mistake) and complain. — sleske, Apr 20 '17 at 15:53

Apache http client defaults don't work with sni

1 Answers1

Linked