C client with OpenSSL + Java server : javax.net.ssl.SSLHandshakeException: no cipher suites in common

Question

I have Java SSL socket and c client with OpenSSL (java clients works ok with this Java server). Handshake fails and i'm getting Java exception:

javax.net.ssl.SSLHandshakeException: no cipher suites in common
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269)
    at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:901)
    at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:629)
    at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:837)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:889)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at EchoServer.main(EchoServer.java:36)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Here is how server SSL socket is created:

public class EchoServer {
    public static void main(String[] arstring) {
        try {
            final KeyStore keyStore = KeyStore.getInstance("JKS");

            final InputStream is = new FileInputStream("/Path/mySrvKeystore.jks");
            keyStore.load(is, "123456".toCharArray());
            final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory .getDefaultAlgorithm());
            kmf.init(keyStore, "123456".toCharArray());
            final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory .getDefaultAlgorithm());
            tmf.init(keyStore);

            SSLContext sc = SSLContext.getInstance("TLSv1.2");
            sc.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new java.security.SecureRandom());

            SSLServerSocketFactory sslserversocketfactory = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
            SSLServerSocket sslserversocket = (SSLServerSocket) sslserversocketfactory.createServerSocket(9997);
            SSLSocket sslsocket = (SSLSocket) sslserversocket.accept();
            sslsocket.setEnabledCipherSuites(sc.getServerSocketFactory().getSupportedCipherSuites());

            InputStream inputstream = sslsocket.getInputStream();
            InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
            BufferedReader bufferedreader = new BufferedReader(inputstreamreader);

            String string = null;
            while ((string = bufferedreader.readLine()) != null) {
                System.out.println(string);
                System.out.flush();
            }
        } catch (Exception exception) {
            exception.printStackTrace();
        }
    }
}

C client:

BIO *certbio = NULL;
BIO *outbio = NULL;
SSL_METHOD *ssl_method;
SSL_CTX *ssl_ctx;
SSL *ssl;

int sd;

// These function calls initialize openssl for correct work.
OpenSSL_add_all_algorithms();
ERR_load_BIO_strings();
ERR_load_crypto_strings();
SSL_load_error_strings();

// Create the Input/Output BIO's.
certbio = BIO_new(BIO_s_file());
outbio  = BIO_new_fp(stdout, BIO_NOCLOSE);

// initialize SSL library and register algorithms
if(SSL_library_init() < 0)
    BIO_printf(outbio, "Could not initialize the OpenSSL library !\n");

ssl_method = (SSL_METHOD*)TLSv1_2_method();

// Try to create a new SSL context
if ( (ssl_ctx = SSL_CTX_new(ssl_method)) == NULL)
    BIO_printf(outbio, "Unable to create a new SSL context structure.\n");

// flags
SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_COMPRESSION | SSL_OP_CIPHER_SERVER_PREFERENCE);

// Create new SSL connection state object
ssl = SSL_new(ssl_ctx);

// Make the underlying TCP socket connection
struct sockaddr_in address;

memset(&address, 0, sizeof(address));
address.sin_family = AF_INET;
address.sin_port = htons(port);

const char *dest_url = this->host.c_str();

address.sin_addr.s_addr = inet_addr(dest_url);
address.sin_port = htons(port);

sd = socket(AF_INET, SOCK_STREAM, 0);
int connect_result = ::connect(sd, (struct sockaddr*)&address, sizeof(address));
if (connect_result != 0) {
    BIO_printf(outbio, "Failed to connect over TCP with error %i\n", connect_result);
    throw IOException("Connection refused");
} else {
    BIO_printf(outbio, "Successfully made the TCP connection to: %s:%i\n", dest_url, port);
}

// Attach the SSL session to the socket descriptor
SSL_set_fd(ssl, sd);

// Try to SSL-connect here, returns 1 for success
int ssl_connect_result = SSL_connect(ssl);
if (ssl_connect_result != 1)
    BIO_printf(outbio, "Error: Could not build a SSL session to: %s:%i with error %i\n", dest_url, port, ssl_connect_result);
else
    BIO_printf(outbio, "Successfully enabled SSL/TLS session to: %s\n", dest_url);

Here is output on client side:

Error: Could not build a SSL session to: 127.0.0.1:9997 with error -1

Update 1

int ssl_connect_result = SSL_connect(ssl);
if (ssl_connect_result != 1) {
    int error_code = SSL_get_error(ssl, ssl_connect_result); // =1
    BIO_printf(outbio, "Error: Could not build a SSL session to: %s:%i with error %i (%i)\n", dest_url, port, ssl_connect_result, error_code);
} else {
    BIO_printf(outbio, "Successfully enabled SSL/TLS session to: %s\n", dest_url);
}

And the output is:

Error: Could not build a SSL session to: 127.0.0.1:9997 with error -1 (1)

Update 2

I forgot to note that I'm using self-signed certificate, generated by keytool from JDK.

Update 3

I've noted i missed some lines and I've added:

OpenSSL_add_all_ciphers();
OpenSSL_add_all_digests();

but still no luck - getting the same error -1.

Update 4

Here is Java client which is accepted by the server code above:

SSLSocketFactory sslsocketfactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket sslsocket = (SSLSocket) sslsocketfactory.createSocket(ip, port);
sslsocket.setEnabledCipherSuites(sslsocketfactory.getSupportedCipherSuites());

InputStream inputstream = System.in;
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);

OutputStream outputstream = sslsocket.getOutputStream();
OutputStreamWriter outputstreamwriter = new OutputStreamWriter(outputstream);
String string = null;

outputstreamwriter.write("hello");
outputstreamwriter.flush();

while ((string = bufferedreader.readLine()) != null) {
    outputstreamwriter.write(string);
    outputstreamwriter.flush();
}
sslsocket.close();

I've checked that I can't see plain data in packets intercepted in the network so it does perform some data encryption.

Answer 1

I don't believe that Java server accepts a Java client either, unless the Java client similarly does .setEnabledCipherSuites (all-supported) -- if so it is using an anonymous (unauthenticated) ciphersuite that is not secure against active attack, and although many people are still stuck in the passive-only threat model of about 1980, today active attacks are common. That is why JSSE's default cipherlist excludes anonymous ciphers -- unless you override it. And why OpenSSL's default cipherlist also excludes them -- which you didn't override.

(add) To explain in smaller words, anonymous ciphersuites ARE encrypted (with a few exceptions not relevant here because they are never preferred) but NOT authenticated. The word "unauthenticated" means "not authenticated"; it does not mean "not encrypted". The word "unencrypted" is used to mean "not encrypted". "Not encrypted" means something that just looks at the channel, like Wireshark, can see the plaintext. "Not authenticated" means an attacker who intercepts (possibly diverts) your traffic can cause you to establish your "secure" session with the attacker in the middle, and they can decrypt your data, copy and/or change it as they wish, re-encrypt it, and send it on, and you will think it is correct and private when it isn't. Google or search here (I think mostly security.SE and superuser) for things like "man in the middle attack", "ARP spoof", "MAC spoof", "DNS poisoning", "BGP attack", etc.

The immediate problem is you aren't using the keystore. You create an SSLContext with key and trust managers from it, but then you create the socket from SSLServerSocketFactory.getDefault() which doesn't use the context. Use sc.getServerSocketFactory() instead.

(add) why? Every SSLSocket (and SSLServerSocket) is linked to an SSLContext which among other things controls the privatekey(s) and certificate(s) or chain(s) used and certificates trusted. (SSL/TLS connections normally authenticate the server only, so in practice the server only needs a key-and-chain, and the client only needs the root cert, but Java uses the same keystore file format for both and it's easy to just code both.) Since your code has set the particular SSLContext sc to contain a suitable key-and-cert, sc.getServerSocketFactory() creates a factory which creates an SSLServerSocket which in turn creates an SSLSocket (for each connection if more than one) which uses that key-and-(as long as the client's supported cipher list allows it, and here it does).

(add) SSLServerSocketFactory.getDefault() creates a factory, and thus sockets, using the default SSL context, which by default contains NO key-and-chain, although you can change this with system properties as described in the documentation cleverly hidden at http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html . As a result it cannot negotiate a ciphersuite that is authenticated. Since both Java and OpenSSL by default disable unauthenticated ciphersuites, this is why you get "no cipher suites in common" unless you .setEnabled to include the unauthenticated and insecure ciphersuites in Java, and still get it for OpenSSL client since you didn't do anything to enable unauthenticated and insecure ciphersuites there.

(add) If you look carefully at your Wireshark trace, you will see in the ServerHello that the selected ciphersuite uses DH_anon or ECDH_anon key exchange -- "anon" is an abbreviation for "anonymous" which means "not authenticated" as explained above -- and there is no Certificate message from the server, and (less obvious unless you know it) the ServerKeyExchange data is not signed.
Also I predict if you have your Java client check sslsocket.getSession().getCipherSuite() and/or sslsocket.getSession().getPeerCertificates() after the handshake is done, which since you don't do it explicitly will be on the first socket-level I/O which will be the outputstreamwriter.flush(), you will see the anonymous ciphersuite, and no peer cert (it throws PeerNotAuthenticated).

Other points:

(1) In general whenever you get SSL_ERROR from SSL_get_error(), or any error return from lower level routines like EVP_* and BIO_*, you should use the ERR_* routines to get details of the error and log/display them; see https://www.openssl.org/docs/faq.html#PROG6 and https://www.openssl.org/docs/manmaster/crypto/ERR_print_errors.html et amici. Especially since you HAVE loaded the error strings, thus avoiding https://www.openssl.org/docs/faq.html#PROG7 . In this case however you already know enough from the server side, so client side details aren't needed.

(2) You don't need _add_all_ciphers and _add_all_digests, they are included in _add_all_algorithms.

(3) OP_NO_SSLv2 and 3 have no effect on TLSv1_2_method and OP_SERVER_CIPHER_PREFERENCE has no effect on a client. (They do no harm, they are just useless and possibly confusing.)

(4) Once you get past the cipher negotiation, the OpenSSL client will need the root cert for the server; since you intend to use a self-signed cert (once you fix the server to use the keystore at all) that cert is its own root. In 1.0.2 (not earlier) you could also use a non-root trust anchor, but not by default and you don't have one anyway. I assume certbio was intended for this, but you never open it on an actual file or do anything else with it, and anyway the SSL library cannot use a BIO for its truststore. You have three choices:

• put the cert(s) in a file, or a directory using special hash names, and pass the file and/or directory name(s) to SSL_CTX_load_verify_locations. If you only want one root (your own) using the CAfile option is easier.

• put or add the cert(s) to the default file or hashed directory determined by your OpenSSL library compilation and call SSL_CTX_set_default_verify_paths; this is commonly something like /etc/pki or /var/ssl. If you want to use the same cert(s) for multiple programs or for commandline openssl this shared location is usually easier.

• use the BIO and/or other means to (open and) read the cert(s) into memory, build your own X509_STORE containing them, and put that in your SSL_CTX. This is more complicated, so I won't expand on it unless you want to.

(5) Your dest_url is (at least in this case?) an address string, not a URL; those are different though related things and thinking they are the same will cause you lots more problems. For most programs it is better to handle a host name string with classic gethostbyname and fall back to inet_addr, or better the "new" (since 1990s) getaddrinfo which can handle both name and address strings and both IPv4 and v6 (also new since 1990s but finally gaining traction). At the very least you should check for inet_addr returning INADDR_NONE indicating it failed.

Answer 2

SSLContext sc = SSLContext.getInstance("TLSv1.2");

Java used to do two thing with code similar to this:

• enable SSLv3
• disable TLS 1.1 and 1.2

... Even though you called out TLS. Effectively, all you could get was a SSLContext with SSLv3 and TLS 1.0 (unless you were willing to do more work). It may not be the case anymore, but it would explain the error you are seeing (especially if your are using Java 7 or 8).

You need to do more work in Java to get "TLS 1.0 or above" and "just TLS 1.2". For that, see Which Cipher Suites to enable for SSL Socket?. It shows you how to enable/disable both protocols and cipher suites.

---

SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_COMPRESSION | SSL_OP_CIPHER_SERVER_PREFERENCE);

You should also set a cipher suite list since OpenSSL includes broken/weak/wounded ciphers by default. Something like:

const char PREFERRED_CIPHERS[] = "HIGH:!aNULL:!kRSA:!PSK:!SRP:!MD5:!RC4";
long res = SSL_CTX_set_cipher_list(ssl_ctx, PREFERRED_CIPHERS);
ASSERT(res == 1);

---

I think your final step is to ensure the self-signed certificate is trusted.

For a Java client, see How to properly import a selfsigned certificate into Java keystore that is available to all Java applications by default?. For an OpenSSL client, just load it up using SSL_CTX_load_verify_locations, see SSL_CTX_load_verify_locations Fails with SSL_ERROR_NONE.

---

Note well: OpenSSL prior to 1.1.0 did not perform hostname validation. You will have to do that yourself. If you need to perform hostname validation, then lift the code from cURL's url.c. I seem to recall looking at Daniel's code, and its very solid. You can also lift the code from Google's CT project. I never audited it, so I don't know what its like.

There are two other checks you have to make; they are discussed at SSL/TLS Client on the OpenSSL wiki.