JDK8 -> JDK10: PKIX path building failed: SunCertPathBuilderException: unable to find valid certification path to requested target

Question

Problem

I have a SpringBoot application using an app called Launchdarkly, which makes use of okhttp
I'm migrating from JRE 8 to JRE 10, calls to other resources work, but fails in calls made using okhttp

EDIT: This could happen with any app that has a certificate chain similar to the one used by our app.

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Exception

The error occurs in thie thread...

config-server_1  | 2018-11-10T21:25:19,147 67327 | DEBUG | okhttp-eventsource-[] ["okhttp-eventsource-stream-[]-0" {}] Connection problem.
config-server_1  | javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
config-server_1  |  at sun.security.ssl.Alerts.getSSLException(Alerts.java:198) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1974) ~[?:?]
config-server_1  |  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:345) ~[?:?]
config-server_1  |  at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:339) ~[?:?]
config-server_1  |  at sun.security.ssl.ClientHandshaker.checkServerCerts(ClientHandshaker.java:1968) ~[?:?]
config-server_1  |  at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1777) ~[?:?]
config-server_1  |  at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:264) ~[?:?]
config-server_1  |  at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1098) ~[?:?]
config-server_1  |  at sun.security.ssl.Handshaker.processRecord(Handshaker.java:1026) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.processInputRecord(SSLSocketImpl.java:1137) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1074) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1402) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1429) ~[?:?]
config-server_1  |  at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
config-server_1  |  at com.launchdarkly.shaded.okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:281) ~[launchdarkly-client-2.3.2.jar!/:2.3.2]
config-server_1  |  at com.launchdarkly.shaded.okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251) ~[launchdarkly-client-2.3.2.jar!/:2.3.2]
config-server_1  |  at com.launchdarkly.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151) ~[launchdarkly-client-2.3.2.jar!/:2.3.2]
config-server_1  |  at com.launchdarkly.shaded.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195) ~[launchdarkly-client-2.3.2.jar!/:2.3.2]
config-server_1  |  at com.launchdarkly.shaded.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121) ~[launchdarkly-client-2.3.2.jar!/:2.3.2]

Setup

Using jlink and selecting modules to build a small JRE
- Installed using Docker at https://dev.to/gimlet2/dockerizing-java-10-spring-boot-app-3b4c
- Current app runs on JRE8 in Docker (same base image)
I only have JAVA_HOME set... Not sure if we need anything else

Java 10 version details

Installed using the method above

root@e0776fd790e7:/runtime# ls -la /etc/ssl/certs/java/cacerts
-rw-r--r-- 1 root root 177280 Oct 29 16:29 /etc/ssl/certs/java/cacerts
root@e0776fd790e7:/runtime# java -version
openjdk version "10" 2018-03-20
OpenJDK Runtime Environment 18.3 (build 10+46)
OpenJDK 64-Bit Server VM 18.3 (build 10+46, mixed mode)

Keystore is set

The java 10 keystore can see it

root@17000659d1ec:/runtime# keytool -cacerts -list
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 80 entries

This is as described at https://dzone.com/articles/openjdk-10-now-includes-root-ca-certificates

Attempts

I'd like to learn if there's anything you can help
- Coming from Unable to find valid certification path to requested target - error even after cert imported
Tried to call cacerts from java 8 to java 10
Tried to symlink or copy cacerts from JRE dir to /etc/ssl/certs/java/cacerts WORKS! ALONG WITH COPYING FROM JDK 8
Tried setting -Djavax.net.ssl.trustStore=/opt/jdk-minimal/jre/lib/security/cacerts

EDIT: See my answer

I would avoid Java 10 as it us eol. Java 11 shouldn't be much harder to migrate too. — Peter Lawrey, Nov 11 '18 at 10:34
@PeterLawrey yeah definitely... Just keeping this documented while we walk towards it...! — Marcello DeSales, Nov 11 '18 at 17:24

score 2 · Accepted Answer · answered Nov 11 '18 at 07:59

Solution when migrating from JDK 8 to JDK 10

The certificates are really different
- JDK 10 has 80, while JDK 8 has 151
JDK 10 has been recently added the certs
- https://dzone.com/articles/openjdk-10-now-includes-root-ca-certificates
- http://openjdk.java.net/jeps/319

JDK 10

root@c339504909345:/opt/jdk-minimal/jre/lib/security #  keytool -cacerts -list
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 80 entries

JDK 8

root@c39596768075:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts #  keytool -cacerts -list
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 151 entries

Steps to fix

I haven't checked which certificate chain is not being trusted, but the server's URL certs are valid... The cacerts from JDK 10 has a chain that's broken as of Today. I can assert that because the download from https://download.java.net/java/GA/jdk10/10/binaries/openjdk-10_linux-x64_bin.tar.gz is being installed in a brand new Docker Image.

I deleted the JDK 10 cert and replaced it with the JDK 8
Since I'm building Docker Images, I could quickly do that using Multi-stage builds
- I'm building a minimal JRE using jlink as /opt/jdk/bin/jlink \ --module-path /opt/jdk/jmods...

So, here's the different paths and the sequence of the commands...

# Java 8
COPY --from=marcellodesales-springboot-builder-jdk8 /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts /etc/ssl/certs/java/cacerts

# Java 10
RUN rm -f /opt/jdk-minimal/jre/lib/security/cacerts
RUN ln -s /etc/ssl/certs/java/cacerts /opt/jdk-minimal/jre/lib/security/cacerts