JAVA Certificates - Programmatically SSL handshake fails and import fails

Question

I wrote a routine for the programmatic retrieval of certificates for SSL connections (to be used for example with restTemplate) and the programmatic import of these certificates to import them into cacerts of the currently used jre.

What I would like to replicate programmatically is the following manual procedure:

keytool -printcert -sslserver {host}:{port} -rfc >> {host}.crt
keytool -importcert -alias {host} -keystore /usr/local/openjdk-8/jre/lib/security/cacerts -file {host} -storepass changeit -noprompt

This is the code I use:

InputStream in = new FileInputStream(keystore);
KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());
ks.load(in, passphrase);
in.close();
 
SSLContext context = SSLContext.getInstance("TLS");

TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
tmf.init(ks);
X509TrustManager defaultTrustManager = null; //(X509TrustManager) tmf.getTrustManagers()[0];
TrustManager[] tms = tmf.getTrustManagers();
for (TrustManager tm : tms) {
    if (tm instanceof X509TrustManager) {
        defaultTrustManager = (X509TrustManager) tm;
        break;
    }
}
if (defaultTrustManager == null) {
    if (isLog) log.error("No Trust Manager found!");
    return false;
}

SaveTrustManager tm = new SaveTrustManager(defaultTrustManager);

context.init(null, new TrustManager[]{tm}, null);

SSLSocketFactory factory = context.getSocketFactory();

if (isLog) log.info("Opening connection -> {}:{} ...", host, port);
 
// Initiate socket
SSLSocket socket = null;
Socket proxiedSocket = null;
if (proxyHost != null) {
    if (isLog) log.info("Using Proxy -> {}:{} ...", proxyHost, proxyPort);

    try {
        if (proxyUser == null || proxyPass == null) {
            Proxy prx = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
            proxiedSocket = new Socket(prx);
            proxiedSocket.connect(new InetSocketAddress(host, port));
            socket = (SSLSocket) factory.createSocket(proxiedSocket, host, port, true);
        } else {
            String proxyUserPass = String.format("%s:%s", proxyUser, proxyPass);                        
            String proxyConnect = "CONNECT " + host + ":" + port + " HTTP/1.0\r\n"
                     + "Proxy-Authorization: Basic "
                        + Base64.getEncoder().encodeToString(proxyUserPass.getBytes(ASCII7)).replace("\r\n", "") + "\r\n"
                     + "Connection: close\r\n"
                     + "\r\n";
            if (isLog) log.info("Socket: {}", proxyConnect);
            
            proxiedSocket = new Socket(proxyHost, proxyPort);
            proxiedSocket.getOutputStream().write(proxyConnect.getBytes(ASCII7));
            proxiedSocket.getOutputStream().flush();
            
            this.readSocketConnection(proxiedSocket);
            
            proxiedSocket.getOutputStream().close();
            
            socket = (SSLSocket) factory.createSocket(proxiedSocket, host, port, true);
        }
    } catch (Throwable t) {
        t.printStackTrace();
        if (isLog) log.info("Proxy Error: {}", t.getMessage());
    }
} else {
    socket = (SSLSocket) factory.createSocket(host, port);
}

socket.setSoTimeout(10000);
try {
    if (isLog) log.info("Initiating SSL handshake...");
    socket.startHandshake();
    socket.close();
    if (proxiedSocket != null) proxiedSocket.close();
    if (isLog) log.info("Certificate is already trusted...");
} catch (SSLException e) {
    //e.printStackTrace(System.out);
    success = false;
    if (isLog) log.warn("Socket Warning: {}", e.getMessage());
}
 
X509Certificate[] chain = tm.chain;
if (chain == null) {
    log.error("Could not obtain server certificate chain");
    return false;
}

if (isLog) log.info("Server sent " + chain.length + " certificate(s):");
 
MessageDigest sha1 = MessageDigest.getInstance("SHA1");
MessageDigest md5 = MessageDigest.getInstance("MD5");
for (int i = 0; i < chain.length; i++) {
    X509Certificate cert = chain[i];
    
    try {
        cert.checkValidity();
        if (isLog) log.info("\tValid certificate...");
    } catch (Throwable t) {
        log.info("\tCertificate invalid!");
        continue;
    }
    
    sha1.update(cert.getEncoded());
    md5.update(cert.getEncoded());
    if (isLog) {
        log.info("\tIndex: {} Subject: {}", (i + 1), cert.getSubjectDN());
        log.info("\tIssuer: {}", cert.getIssuerDN());
        log.info("\tSHA-1: {}", toHexString(sha1.digest()));
        log.info("\tMD5: {}", toHexString(md5.digest()));
    }
    
    String alias = host + "-" + (i + 1);
    ks.setCertificateEntry(alias, cert);

    if (isLog) {
        log.info("{}", cert);
        log.info("Added certificate to keystore '{}' using alias '{}'", keystore, alias);
    }
    
}

OutputStream out = new FileOutputStream(keystore);
ks.store(out, passphrase);
out.close();

On one particular VM I get this error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The weird thing is that locally it works and it works on other VMs too. Also the other strange thing is that if I convert the downloaded certificate (although the handshake fails) in PEM format (base64) and I compare it with the one exported by keytool it's identical!

Also even if I try to load the downloaded PEM file with keytool using the Keystore programmatic procedure in java, I don't get any errors, but when I go to use a restTemplate to make that call.., I get the same error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

In particular this VM does not run behind a proxy...

I did a lot of tests and I tried with -Djavax.net.debug=ssl,handshake to get some more information, but unfortunately I didn't come up with it.

I tried importing the PEM programmatically from a keytool downloaded file manually and got the same error. I tried to verify to reach the portal in https with wget/curl/openssl and I reach it correctly. There are no proxies behind the VM. I tried giving full access (chmod) to cacerts file.

The portal only downloads 1 certificate from the chain of the 3 certificates but it is the same one that it downloads with keytool (the PEMs downloaded with keytool and the one downloaded programmatically are identical).

Everything is dockerized and runs under tomcat:9.0.35-jdk8-openjdk. On other machines and locally it works correctly even behind proxies.

The VM it doesn't work on is an Ubuntu 20 (Debian).

Answer 1

I think I solved the issue.

I have a routine that programmatically downloads certificates from SSL protected servers through a socket that handshake and retrieve certificates. Currently the certificate SSL server has some problems in returning the whole chain of certificates (including the intermediary certificates of the CAs that signed the certificate). It is likely that they are problems related to a misconfiguration.

This problem can be verified with the command: openssl s_client -showcerts -connect host:port These are the intermediary CAs irrecoverable certs: certs certs

These are the intermediary certificates of the CAs: Intermediate cert CAs chain

On a programmatic level, when JAVA persists the trusted certificates in the Trust-Store, it writes information related to the fact that it was not possible to recover the whole CA chain. It is as if the certificate is self-signed for the trust-store when it is not. In fact, the certificate is not actually trusted (unless you enter it "manually" in the trust store). When making an external call to an application-side https server, the client is not fully aware of the certificate. The error was: "PKIX path building failed". It was necessary to use a custom built HTTP client to accept single certificates of which it was not possible to download the whole CA chain. Obviously it is not a client that can accept self-signed certificates. This client is "satisfied" with the single certificate but it is actually a necessary and sufficient condition to perform a secure SSL handshake.

In other words to make an external call with RestTemplate what you have to do is change it from:

restTemplate = new RestTemplate();

to:

restTemplate = new RestTemplate();
HttpClient httpClient = HttpClients.custom().build();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory(httpClient));

I think I have given an explanation. What do you think about it?