1

I'm running a local TCP server in C++ on Windows via OpenSSL and Windows sockets. I'm running the client in Java using SSL sockets. For most users this setup is working, however, some users run into the following Java exception upon attempting the SSL handshake:

javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
    at sun.security.ssl.SSLSocketImpl.handleEOF(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketImpl.decode(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source) ~[?:1.8.0_321]
    ...
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source) ~[?:1.8.0_321]
    at sun.security.ssl.SSLTransport.decode(Unknown Source) ~[?:1.8.0_321]
    ... 10 more

My client socket code in Java looks like this:

public static Socket getClientSocket() throws Exception
{
    InetSocketAddress socketAddress = new InetSocketAddress("localhost", 54321);
    SSLSocketFactory socketFactory = getSSLSocketFactory();
    Socket clientSocket = socketFactory.createSocket();
    clientSocket.connect(socketAddress, 1_000);
    ((SSLSocket) clientSocket).startHandshake(); // <-- Exception here for some users
    return clientSocket;
}

private static SSLSocketFactory getSSLSocketFactory() throws Exception
{
    // Create a trust manager that does not validate certificate chains
    X509TrustManager trustManager = new X509TrustManager()
    {
        @Override
        public X509Certificate[] getAcceptedIssuers()
        {
            // TODO Returning null is not allowed but it still works
            return null;
        }

        @Override
        public void checkClientTrusted(final X509Certificate[] certificates, final String authType)
        {

        }

        @Override
        public void checkServerTrusted(final X509Certificate[] certificates, final String authType)
        {
            for (X509Certificate certificate : certificates)
            {
                String certificateString = certificate.toString();
                if (!certificateString.contains("blablabla"))
                {
                    throw new SSLException("Certificate not trusted");
                }
            }
        }
    };
    SSLContext sslContext = SSLContext.getInstance("SSL");
    sslContext.init(null, new TrustManager[]{trustManager}, new SecureRandom());
    return sslContext.getSocketFactory();
}

Attempted solutions:

  • Calling System.setProperty("https.protocols", "TLSv1,TLSv1.1,TLSv1.2") before connecting did not work (reference)
  • I do not want to tell users to import any SSL certificates into their truststore (unless I can automate this easily)

Where does the inconsistent behavior come from and how to fix it cleanly for everyone who gets the exception? I have access to the server source code and certificates. Also any critical questions or suggestions are welcome. I can provide more information if necessary.

BullyWiiPlaza
  • 17,329
  • 10
  • 113
  • 185
  • Could it be that some of your clients have an old Java version that still allows the now deprecated TLSv1? – RealSkeptic Jan 24 '22 at 10:29
  • @RealSkeptic: Maybe, so forcing e.g. `TLSv1.2` would help, e.g. like this answer? https://stackoverflow.com/a/42291244/3764804 – BullyWiiPlaza Jan 24 '22 at 10:52
  • You can try that. – RealSkeptic Jan 24 '22 at 11:18
  • @RealSkeptic: I did it and they still get the same exception. Maybe there is something I don't understand about SSL, what else could it be? Their browsers also work fine with HTTPS I assume. I'm disabling the trust manager in Java so that can't be it either. – BullyWiiPlaza Jan 24 '22 at 11:20
  • Perhaps their Java doesn't support TLS v1.2. Check the version. That error is most commonly caused because the server and the client don't speak the same protocol. The browser is irrelevant. – RealSkeptic Jan 24 '22 at 11:25
  • @RealSkeptic: Why would Java not support it these days? It even occurred with Java 17 so that didn't fix it either. – BullyWiiPlaza Jan 24 '22 at 11:35
  • 1
    If you can start your client with JVM option `-Djavax.net.debug=all` you can find the reason in the SSL debug output. – Stefan D. Jan 24 '22 at 12:18
  • `I do not want to tell users to import any SSL certificates into their truststore` You can create an empty java keystore, import the required server certificate(s) and use this in the client SSL setup. This truststore can be deployed/rolled out together with the application. – Stefan D. Jan 24 '22 at 12:27

0 Answers0