Communication issue between Spring Boot application (JHipster) and Keycloak through HTTPS

Question

In my team, we're trying to deploy a microservice stack based on JHipster (6.8.0) on OpenShift (4.2).

We have currently an issue when the gateway starts and tries to communicate with Keycloak through HTTPS (using Red Hat Single Sign On 7.3 based on Keycloak to be precise).

Here is the exception that is raised:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

We think it is because our gateway doesn't trust certificates from Keycloak. Indeed, this one is using organization certificates. We have logged in to the Keycloak administration interface on the realm the gateway is trying to connect with. And we have extracted certificates as X.509 binary-encoded DER files; thanks to Chrome browser functionality.

We first tried to simply add these certificates to the etc/ssl/certs/java/cacerts folder of our gateway container. To do so, we created these folders in the project jib repository, src/main/jib/etc/ssl/certs/java/cacerts, and copied the certificates to it.

We have generated our gateway Docker image using Maven and the jib:dockerBuild option. We pushed it to our Docker registry and deployed it to OpenShift. After a check in the OpenShift pod, certificates are well in place in etc/ssl/certs/java/cacerts. But we still get the same error as before.

We then tried to use a truststore. So we created one using this command for each certificate:

keytool -import -file path/to/certificate.cer -alias certificateAlias -keystore applicationTrustStore.jks

We checked that all certificates were properly added, thanks to this command:

keytool -list -v -keystore applicationTrustStore.jks

Then we added this applicationTrustStore.jks file into the src/main/jib/etc/ssl/certs/java/cacerts folder of our project and added this in the pom.xml:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <version>${jib-maven-plugin.version}</version>
    <configuration>
        <from>
            <image>adoptopenjdk:11-jre-hotspot</image>
        </from>
        <to>
            <image>application:latest</image>
        </to>
        <container>
            …
            <jvmFlags>
                <jvmFlag>-Djavax.net.ssl.trustStore=etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
                <jvmFlag>-Djavax.net.ssl.trustStoreType=jks</jvmFlag>
                <jvmFlag>-Djavax.net.ssl.trustStorePassword=password</jvmFlag>
            </jvmFlags>
        </container>
        …
    </configuration>
</plugin>

Once again, we have generated and redeployed to OpenShift with no luck; still exactly the same issue.

We're certainly missing something obvious but we can't put our finger on it.

Just in case, `applicationTrustStore.jks` should be based on a working `cacerts`. That is, in addition to your certs, it should contain all the root CA certs. — Chanseok Oh, Oct 11 '20 at 12:38
Hello, thanks for your comment ! Will try to build another truststore from a working one and will make a try. — Ajrarn, Oct 13 '20 at 03:41

deduper · Answer 1 · 2020-10-11T10:08:06.673

TL;DR — Try adding the certificate to the cacerts keystore of the JRE on which your application's Docker image is based.

The long-winded answer

Ahh yes! The old …PKIX path building failed… problem. Been there. And OpenShift was also involved to boot, in one instance.

Having come across that same error maybe a dozen or so times at this point, I would bet that the reason why your efforts so far had no effect, is because that javax.net.ssl.SSLHandshakeException is coming from the JRE running your application; not from whatever it is at /etc/ssl/certs/java/.

The solution

From your browser, obtain the Root CA certificate in question (what you referred to as: „…the Realm the gateway is trying to connect with…“)
- Extract it the way you said you did in your question („…as X-509 binary encoded DER files…“)
Add the certificate to the cacerts keystore of the JRE on which your application's Docker image is based (<image>adoptopenjdk:11-jre-hotspot</image>)
- An example Dockerfile to give you an idea of what you'll need to do:¹

FROM adoptopenjdk:11-jre-hotspot

COPY  stackexchange.cer /tmp/cert/certificate.cer

RUN  keytool -noprompt -import -alias deduper.answer -storepass changeit -keystore /opt/java/openjdk/lib/security/cacerts -file /tmp/cert/certificate.cer

CMD ["keytool", "-list", "-keystore",  "/opt/java/openjdk/lib/security/cacerts", "-alias", "deduper.answer", "-storepass", "changeit" ]

You can pull from Docker Hub, the image that Dockerfile builds…

$ docker pull deduper/ajrarn.soq.pkix.fix

Run it to observe that the cert with the alias was added…²

$ docker run -it deduper/ajrarn.soq.pkix.fix

Warning: use -cacerts option to access cacerts keystore
deduper.answer, Oct 10, 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): E5:81:5A:DF:11:A9:0C:CC:51:8F:6A:99:D2:6C:67:16:29:D6:68:E1:EA:C2:C0:A7:E7:9B:84:09:AF:9C:29:14

If even that doesn't solve your problem, then the next thing I would suggest is to change this…

…
<jvmFlag>-Djavax.net.ssl.trustStore=etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
…

To this…

…
<jvmFlag>-Djavax.net.ssl.trustStore=/etc/ssl/certs/java/cacerts/applicationTrustStore.jks</jvmFlag>
…

In case you didn't spot the difference, you've left out the leading forward slash from: /etc/…

^¹ ^{You'll have to adapt however you build your image to jibe with the instructions in the example Dockerfile.}
^² ^{I used the Stack Overflow cert for this experiment.}

A huge thanks @deduper for your very clear and detailed answer and to have taken time to load a Docker image ! — Ajrarn, Oct 13 '20 at 03:31
I first tried the easiest way and update the /etc/ part but it didn't change the issue. I then try to follow your example (using jib in Maven) with keytool import command on /opt/java/openjdk/lib/security/cacerts. Unfortunately, when trying this I get a Permission denied from Openshift (T_T) If I'm not wrong, by default, root user is not authorized. For now I will try to dig around jvm options for trustStore. Should work right ? Any idea where I'm wrong on my current attempt ? Thanks ! — Ajrarn, Oct 13 '20 at 03:38
Hey, thanks for getting back. — „*…by default, root user is not authorized…*“ – @Ajrarn — Do you mean the *root* user in the *`adoptopenjdk:11-jre-hotspot`*? Or do you mean the *root* user in OpenShift? You most likely mean the latter. Because the *root* user in my experimental [*`deduper/ajrarn.soq.pkix.fix `*](https://hub.docker.com/r/deduper/ajrarn.soq.pkix.fix) image/container is authorized. Have you run that container? The OpenShift project I referred to was 3 years ago. So I don't recall the level of detail re: permissions. As far as „*using jib in Maven*“, I'd have to research it. — deduper, Oct 13 '20 at 12:20
„*…Should work right ?…*“ – @Ajrarn — Right. I've encountered that same „*`javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`*“ error message, verbatim, a dozen times. Each and every time, it was resolved by the command I shared with you in my answer. Now, the ***specific*** details of how *you* will need to apply that proposed solution in *your* system; that's something for *you* to know. If you have additional, more *specific* questions, just let me know. TIA. — deduper, Oct 13 '20 at 13:48
In fact I was getting a _Permission denied_ when using keytool import command in **entrypoint.sh** file defined in my jib configuration. It worked on my workstation but not in Openshift. — Ajrarn, Oct 13 '20 at 15:20
And just for you to know my _Should work right ?_ was in the usage of _-Djavax.net.ssl.trustStore_. But finally I find another that I will explain soon. Another big thank you, your explanations help a lot ! — Ajrarn, Oct 13 '20 at 15:22

score 1 · Answer 2 · answered Oct 12 '20 at 16:37

The JRE in adoptopenjdk:11-jre-hotspot never reads /etc/ssl/certs/java/cacerts by default. Most JREs actually read <JRE>/lib/security/cacerts by default (unless you set -Djavax.net.ssl.trustStore).

It is just that on many Linux distros, often <JRE>/lib/security/cacerts is a symlink to /etc/ssl/certs/java/cacerts. For example,

# ls -l /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts
lrwxrwxrwx    1 root     root            27 Jan  1  1970 /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -> /etc/ssl/certs/java/cacerts

In such cases, placing cacerts at /etc/ssl/certs/java will work. However, in adoptopenjdk:11-jre-hotspot, <JRE>/lib/security/cacerts is not a symlink, as shown below:

# ls -l /opt/java/openjdk/lib/security/cacerts 
-rw-r--r-- 1 root root 101001 Jul 15 09:07 /opt/java/openjdk/lib/security/cacerts

As @deduper explained, I'd put the file into /opt/java/openjdk/lib/security/.

And if you want to set -Djavax.net.ssl.trustStore to specify a different location, I think the path should be an absolute path as @deduper pointed out.

Ajrarn · Answer 3 · 2020-10-14T02:56:24.313

Thanks to the help of @deduper and @Chanseok Oh, I was able to solve the issue.

Few explanations in case some JHipster/Openshift users step around here.

First, I have tried to use keytool command to import organization certificate in /opt/java/openjdk/lib/security/cacerts. So I create tmp/cert subfolder in src/main/jib folder, put organization certificate inside and update src/main/jib/entrypoint.sh file like this:

#!/bin/sh

echo "The application will start in ${JHIPSTER_SLEEP}s..." && sleep ${JHIPSTER_SLEEP}
exec keytool -noprompt -import -alias alias -storepass changeit -keystore /opt/java/openjdk/lib/security/cacerts -file /tmp/cert/organization.cer

exec java ${JAVA_OPTS} -noverify -XX:+AlwaysPreTouch -Djava.security.egd=file:/dev/./urandom -cp /app/resources/:/app/classes/:/app/libs/* "com.your.company.App"  "$@"

It seems to work locally but when starting that on Openshift, I was getting a Permission denied when keytool was called because by default Openshift doesn't use root user. So, I finally remove my modifications and come back to the original entrypoint.sh file.

The workaround I use was:

to extract cacerts file from adoptopenjdk:11-jre-hotspot (using docker cp command, more info: Copying files from Docker container to host)
add certificate inside cacerts using keytool command on my workstation
create these subfolders in src/main/jib: opt/java/openjdk/lib/security
copy cacerts file inside

When creating Docker image using jib, cacerts in opt/java/openjdk/lib/security contains your organizational certificate and when starting on Openshift, communication with Keycloak is ok.

Glad you found the right solution to put the pre-built `cacerts` under `src/main/jib/...`. But I'd like to point out that it's a poor practice to run `keytool` as part of `entrypoint.sh` (i.e., importing at runtime when starting up your container and application). It's better to prepare things at compile time and build a working image; as an analogy, you don't want to run install and set up JDK by `apt-get get install openjdk` in `entrypoint.sh`. I suggest to update your answer to prevent other users attempt setting up things at runtime. — Chanseok Oh, Oct 13 '20 at 20:31

Communication issue between Spring Boot application (JHipster) and Keycloak through HTTPS

3 Answers3