Can't connect to specific https website using WebClient in Java

Question

Summary: I've created a simple block of code that I've used several times in the past with success, but it is not working with this website ("https://mid-usa.com/login") for some reason. I can telnet from my computer to mid-usa.com on 443, and I can open this page in a browser.

    WebClient client = new WebClient(BrowserVersion.BEST_SUPPORTED);
    client.getOptions().setSSLClientProtocols(new String[] { "TLSv1.2","TLSv1.1"});
    client.getOptions().setCssEnabled(false);
    client.getOptions().setJavaScriptEnabled(false);
    client.getOptions().setUseInsecureSSL(true);
    client.getOptions().setTimeout(10000);
    HtmlPage page = (HtmlPage) client.getPage("https://mid-usa.com/login");

The reason I am setting the setSSLClientProtocols to TLSv1.2, is because upon looking at the website's protocols running a scanner, I thought it would help (from reading other posts), but it didn't. I also left it with just TLSv1.2, and still get the following print out:

Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:28 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {s}->https://mid-usa.com:443: Connection reset
Jul 03, 2020 3:23:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://mid-usa.com:443
Connection resetjava.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at sun.security.ssl.InputRecord.readFully(Unknown Source)
    at sun.security.ssl.InputRecord.read(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
    at com.gargoylesoftware.htmlunit.httpclient.HtmlUnitSSLConnectionSocketFactory.connectSocket(HtmlUnitSSLConnectionSocketFactory.java:184)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
    at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:193)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1537)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1456)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:447)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501)
    at ScraperBase.autoLogin(ScraperBase.java:25)
    at MidwayDataCollection$4.actionPerformed(MidwayDataCollection.java:222)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)

Any ideas?

See if this helps https://stackoverflow.com/questions/5507878/ssl-connection-reset — tgdavies, Jul 04 '20 at 05:57
Thank you for the link tgdavies. When searching for my issue, I noticed this and ran a scan and noticed it supported TLSv1.2, which is why i set that in the code. I also tried to use a lower-level library, as in the post, to create the socket manually. I still get a connection reset. I'm not sure why, and I'm out of ideas. The browser works fine though. — dachikzuki, Jul 04 '20 at 23:55
In case of redirects, I did add redirects "client.getOptions().setRedirectEnabled(true);" and I still get the same problem with a connection reset though. I'm just not sure why it's only for this specific website. I ran a scan on it and enabled the protocols. I increased the timeout. I can change the URL to anything else and no problem, yet I can easily hit this page with an actual browser. Any ideas on where I can detect exactly what is special about this website, that I need to overcome? — dachikzuki, Jul 05 '20 at 10:22

tgdavies · Accepted Answer · 2020-07-05T21:19:24.500

I believe that the problem is that the cipher suite the server wants to use is disabled in later JDKs.

When I connected via curl -v https://mid-usa.com >/dev/null I got:

* SSL connection using TLSv1.2 / DES-CBC3-SHA

This maps to SSL_RSA_WITH_3DES_EDE_CBC_SHA in JSSE (https://docs.oracle.com/cd/E19728-01/820-2550/cipher_suites.html)

I rewrote your example to just use a URLConnection:

        URL url = new URL("https://mid-usa.com");
        URLConnection urlConnection = url.openConnection();
        InputStream is = urlConnection.getInputStream();

And ran it with -Djavax.net.debug=all -Dhttps.cipherSuites="SSL_RSA_WITH_3DES_EDE_CBC_SHA"

Ignoring disabled cipher suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA for TLSv1.2
No available cipher suite for TLSv1.2
main, handling exception: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

So I think that the server want to use a cipher that the JDK won't allow.

This Jira issue seems similar https://bugs.openjdk.java.net/browse/JDK-8213846 -- the server drops the connection after the handshake is sent.

So the question now becomes: how to enable that cipher suite, which I have not figured out how to do!

Yes, 8u171 and j10 up disable 3DES ciphersuites by default, in the `jdk.tls.disabledAlgorithms` security property; see https://sweet32.info . This can be changed by (1) edit j8/lib/security/java.security or j9+/conf/security/java.security (2) create editted java.security elsewhere and point to it with sysprop java.security.properties, see https://stackoverflow.com/questions/4521119/ and https://stackoverflow.com/questions/35140079/ (3) call Security.setProperty early in code (before the TLS classes are loaded). SSLabs gives that server a grade of C for this! — dave_thompson_085, Jul 05 '20 at 16:12
You guys are amazing. Thank you so much. I learned a lot from your links and was able to fix the problem. the java.security file has enabled the ability to edit it by default (security.overridePropertiesFile=true). I was able to set the following within my code without having to edit anything in the environmment: `Security.setProperty("jdk.tls.disabledAlgorithms", "RC4, MD5withRSA"); Security.setProperty("jdk.tls.legacyAlgorithms", "TLS_RSA_WITH_3DES_EDE_CBC_SHA"); System.setProperty("https.protocols", "TLSv1,TLSv1.1");` — dachikzuki, Jul 29 '20 at 18:02

Can't connect to specific https website using WebClient in Java

1 Answers1