SSL socket connection with client authentication

Question

I have an application server running some utility commands, which is programmed in C. I have to connect to the server through Java client program using Java SSL socket with client authentication. The key on the server side was created using:

   openssl req -new -text -out ser.req
   openssl rsa -in privkey.pem -out ser.key
   openssl req -x509 -in ser.req -text -key ser.key -out ser.crt

I have been provided the server key and certificate. I have combined the key and certificate into a PKCS12 format file:

openssl pkcs12 -inkey ser.key -in ser.crt -export -out ser.pkcs12

Then loading the resulting PKCS12 file into a JSSE keystore with keytool:

keytool -importkeystore -srckeystore ser.pkcs12 -srcstoretype PKCS12 -destkeystore ser.keystore

But when I try to connect, I get the following error:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
    at SSLSocketClient.main(SSLSocketClient.java:67)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
    at sun.security.validator.Validator.validate(Validator.java:271)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
    ... 11 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)
    ... 17 more

On the server side log:

SSL open_server: could not accept SSL connection: sslv3 alert certificate unknown

Running command:

java -Djavax.net.ssl.keyStore=/path/to/ser.keystore -Djavax.net.ssl.keyStorePassword=passwd SSLSocketClient <server-ip> <port>

Does anyone know the cause of this problem?

Updated the client source code:

import java.net.*;
import java.io.*;
import javax.net.ssl.*;

import java.security.cert.CertificateFactory;
import java.security.cert.Certificate;
import java.security.cert.X509Certificate;
import java.security.KeyStore;
import java.security.SecureRandom;
import javax.net.SocketFactory;

public class SSLSocketClient {

   public static void main(String [] args) throws Exception {
      String serverName = args[0];
      int port = Integer.parseInt(args[1]);
      try {

        SSLSocketFactory sf =
                (SSLSocketFactory)SSLSocketFactory.getDefault();

        Socket client = new Socket(serverName, port);

        System.out.println("Connected to " + client.getRemoteSocketAddress());
        OutputStream outToServer = client.getOutputStream();
        DataOutputStream out = new DataOutputStream(new BufferedOutputStream(outToServer));

        writeData(out);
        out.flush();

        InputStream inFromServer = client.getInputStream();
        DataInputStream in = new DataInputStream(inFromServer);

        
        readData(in);
        outToServer = client.getOutputStream();
        out = new DataOutputStream(new BufferedOutputStream(outToServer));
        writeData2(out);
        out.flush();
        
        Socket newClient = sf.createSocket(client, serverName, port, false);

        client.close();
      } catch (IOException e) {
         e.printStackTrace();
      }
   }

    private static void writeData(DataOutputStream out) throws IOException {
         char CMD_CHAR_U = 'U';
         byte b = (byte) (0x00ff & CMD_CHAR_U);

         out.writeByte(b);          // <U>
    }

    private static void writeData2(DataOutputStream out) throws IOException {
         char CMD_CHAR_S = 'S';
         byte b = (byte) (0x00ff & CMD_CHAR_S);

         out.writeByte(b);          // <S>
    }

    private static void readData(DataInputStream in) throws IOException {
        char sChar = (char) in.readByte(); 
        System.out.println("<S>\t\t" + sChar);
    }
}

Now creating the truststore as shown in the link: https://jdbc.postgresql.org/documentation/head/ssl-client.html

Steps to create:

openssl x509 -in server.crt -out server.crt.der -outform der
keytool -keystore mystore -alias clientstore -import -file server.crt.der
java -Djavax.net.ssl.trustStore=mystore -Djavax.net.ssl.trustStorePassword=mypassword com.mycompany.MyApp

Note - The server side is using TLSv1 protocol

But still not able to make it through. What am I doing wrong? What I want is the server to authenticate the crt of the client.

The login protocol with server; the SSL we use is only to authenticate not to secure the transmission:

    -------------------------------------------------------------
    client                                            server 
    -------------------------------------------------------------

    sock = connect()                                 sock = accept()
                      <U><LOGIN_SSL=501>
                 --------------------------------->
                       'S'|'E'
                 <---------------------------------
                       'S'
                 --------------------------------->
    SSL_connect(sock)                               SSL_accept(sock)

                      <R><LOGIN_SSL>
                 <---------------------------------

Answer 1

I think you have several problems with your setup.

To configure properly the SSL connection with JSSE you need several things depending if you need to authenticate the server, the client, or to perform mutual authentication.

Let's suppose the later and more complete use case of mutual authentication.

The objective is to configure a SSLSocketFactory that you can use to contact your server.

To configure a SSLSocketFactory, you need a SSLContext.

This element in turn with require at least two elements for the mutual authentication use case, a KeyManagerFactory, required for client side SSL authentication, i.e., the server to trust the client, and TrustManagerFactory, required for configuring the client to trust the server.

Both KeyManagerFactory and TrustManagerFactory require a properly configured keystore with the necessary cryptographic material.

So, the first step will consist on generating this cryptographic material.

You already created a keystore with the server certificate:

keytool -keystore serverpublic.keystore -alias clientstore -import -file server.crt.der -storepass yourserverpublickeystorepassword

Please, be aware that, in a similar way as in the server case, you also need to create a public and private key pair for your client, of course, different than the server one.

The related code you provided with OpenSSL and keytool looks appropriate. Please, repeat the process for the client side:

openssl req -new -text -out client.csr
openssl rsa -in clientpriv.pem -out client.key
openssl req -x509 -in client.csr -text -key client.key -out client.crt
// You can use PKCS12 also with Java but it is also ok on this way
openssl pkcs12 -inkey client.key -in client.crt -export -out client.pkcs12
// Do not bother yourself and, in this use case, use always the same password for the key and keystore
keytool -importkeystore -srckeystore client.pkcs12 -srcstoretype PKCS12 -destkeystore client.keystore -storepass "yourclientkeystorepassword"

With the right keystores in place, try something like the following to interact with your server:

// First, let's configure the SSL for client authentication
KeyStore clientKeyStore = KeyStore.getInstance("JKS");
clientKeyStore.load(
  new FileInputStream("/path/to/client.keystore"),
  "yourclientkeystorepassword".toCharArray()
);

KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); // SunX509
kmf.init(clientKeyStore, "yourclientkeystorepassword".toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

// Now, let's configure the client to trust the server
KeyStore serverKeyStore = KeyStore.getInstance("JKS");
serverKeyStore.load(
  new FileInputStream("/path/to/serverpublic.keystore"),
  "yourserverpublickeystorepassword".toCharArray()
);

TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()); // SunX509
tmf.init(serverKeyStore);
TrustManager[] trustManagers = tmf.getTrustManagers();

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null); // You can provide SecureRandom also if you wish

// Create the SSL socket factory and establish the connection
SSLSocketFactory sf = sslContext.getSocketFactory();
SSLSocket socket = (SSLSocket)sf.createSocket(serverName, port);

// Interact with your server. Place your code here
// Please, consider the following link for alternatives approaches on how to 
// interchange information with the server:
// https://web.mit.edu/java_v1.5.0_22/distrib/share/docs/guide/security/jsse/samples/sockets/client/SSLSocketClient.java
// It also suggest the use of startHandshake explicitly if your are using PrintWriter for the reason explained in the example an in the docs:
// https://docs.oracle.com/en/java/javase/11/docs/api/java.base/javax/net/ssl/SSLSocket.html

//...

// Close the socket
socket.close();

The described approach can be extended to use, instead of sockets, higher level of abstraction components like HttpsURLConnection and HTTP clients - with the exception of Apache HttpClient that handles SSL differently - like OkHttp which, under the hood, use SSLSocketFactory and related stuff.

Please, also consider review this great article from IBM's DeveloperWorks, in addition to explain many of the point aforementioned will provide you great guidance with the generation of keystores for your client an server if necessary.

Please, also be aware that, depending on your server code, you may need to configure it to trust the provided client certificate.

According to your comments you are using a server side code similar to the one provided by Postgresql 8.1. Please, see the relevant documentation for configuring SSL in that database, if you are using some similar server side code it maybe could be of help.

Probably the best approach will be to generate a client certificate derived from the root certificate trusted by your server instead of using a self signed one.

I think that it will be also relevant for your server side SSL certificate an associated private key: first, create a root self signed certificate, your CA certificate, configure your server side C code to trust it, and then derive both client and server side SSL cryptographic material from that CA: probably it will simplify your setup and make everything work properly.