Can't connect to broker over SSL

Question

I have an ActiveMQ instance running on localhost that's trying to connect to another instance running on AWS (Amazon MQ). The broker running on AWS requires the broker-to-broker connection to be made over SSL.

In my localhost broker's activemq.xml, I added the following to create a connection with the broker running on AWS (just one way for now):

<networkConnectors>
    <networkConnector name="local-to-aws" userName="myCommonUser" duplex="true" uri="static:(ssl://###.mq.ca-central-1.amazonaws.com:61617)"/>
</networkConnectors>

If I try starting my localhost broker with that, I get the following error in my activemy.log file:

2021-01-29 20:19:32,638 | WARN  | Could not start network bridge between: vm://localhost and:
    ssl://###.mq.ca-central-1.amazonaws.com:61617 due to: java.security.NoSuchAlgorithmException: 
    Error constructing implementation (algorithm: Default, provider: SunJSSE, class: 
    sun.security.ssl.SSLContextImpl$DefaultSSLContext) | 
    org.apache.activemq.network.DiscoveryNetworkConnector | Simple Discovery Agent-2

So to try and fix that, I added the following in the localhost broker's activemq.xml config:

<sslContext>
    <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
                keyStorePassword="###"
                trustStore="file:${activemq.base}/conf/client.ts"
                trustStorePassword="###"/>
</sslContext>

The broker.ks and client.ts were generated using the following commands:

Using keytool, create a certificate for the broker:
    keytool -genkey -alias broker -keyalg RSA -keystore broker.ks

Export the broker's certificate so it can be shared with clients:
    keytool -export -alias broker -keystore broker.ks -file broker_cert

Create a certificate/keystore for the client:
    keytool -genkey -alias client -keyalg RSA -keystore client.ks

Create a truststore for the client, and import the broker's certificate. This establishes that the client "trusts" the broker:
    keytool -import -alias broker -keystore client.ts -file broker_cert

And that's all I did to enable SSL on my localhost broker, and I'm not sure if there's anything else I need to do besides adding that <sslContext> element to my config.

When I try starting the localhost broker after having made that change, I now get a different error in activemq.log:

2021-01-29 20:07:03,686 | INFO  | Error with pending remote brokerInfo on: 
   ssl://###.mq.ca-central-1.amazonaws.com/##.###.#.106:61617 (Connection or inbound has closed) |
   org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ Transport: ssl://###.mq.ca-
   central-1.amazonaws.com/##.###.#.106:61617

2021-01-29 20:07:03,687 | WARN  | Could not start network bridge between: vm://localhost and: 
   ssl://###.mq.ca-central-1.amazonaws.com:61617 due to: sun.security.validator.ValidatorException: 
   PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
   valid certification path to requested target | org.apache.activemq.network.DiscoveryNetworkConnector | 
   Simple Discovery Agent-3

And this is where I'm stumped.

Does anyone know what I'm doing wrong? I don't see anything SSL related in the AWS broker's config, and I'm assuming that's because AWS has intentionally hid that from me to prevent me from changing any of the SSL settings for that broker.

Do I need to somehow get a hold of the SSL cert used by the AWS broker and add that to my localhost broker? I have no clue what to do next.

Update: I used Portecle download the SSL certs from the AWS broker and added them to client.ts.

@Tch Yes, the firewall is configured to allow SSL OpenWire connections from the localhost broker to the AWS broker. I was getting a different error (connection refused) when the firewall hadn't been configured. That is no longer the issue I think. — user928112, Jan 30 '21 at 02:49
did you setup your ip network to the inbound rules on amazonmq console as well? — Tch, Jan 30 '21 at 09:12
@Tch Yes, I did as far as I can tell. The firewall and other aspects of the network are controlled by the network team at my company. All I know is that they're currently allowing inbound and outbound TCP over SSL connections to Amazon MQ on port 61617. It is possible that they may have misconfigured something, but I'd have to double check with them. What makes you think that this could be related to a firewall or some other network related issue? — user928112, Jan 30 '21 at 09:37
can you try run keytool to inspect it and double check the password is correct. Basing this on https://stackoverflow.com/questions/9761575/java-nosuchalgorithmexception-sunjsse-sun-security-ssl-sslcontextimpldefault — Yan, Jan 30 '21 at 14:19
@Yan I will check. Do I also have to add the certificate to "jre/lib/security/cacerts"? Right now I just have the certificate sitting in "file:${activemq.base}/conf/". — user928112, Jan 30 '21 at 18:01
It's also worth verifying that your app's JVM is configured correctly. Follow these steps: https://stackoverflow.com/a/36427118/421195 — paulsm4, Feb 03 '21 at 22:38
Did you trust the starfield CA on your local broker? Does the remote broker trust the certificate chain that you created for your local broker? — Khanna111, Feb 03 '21 at 22:40
@Khanna111 I just used Portecle to download that cert and 3 more that showed up (there were 4 in total). How do I add these to client.ts? — user928112, Feb 03 '21 at 22:55
open up truststore file and for the time being just upload all these certs into the truststore as trusted certifcates. https://www.rgagnon.com/javadetails/java-fix-certificate-problem-in-HTTPS.html — Khanna111, Feb 03 '21 at 23:52
if it works, then we can clean up and add only the necessary one. — Khanna111, Feb 03 '21 at 23:52
@user928112: original question has been answered. Please mark the answer as accepted and ask the next question with a snapshot of the remote broker log. — Khanna111, Feb 04 '21 at 04:57

score 1 · Accepted Answer · answered Feb 04 '21 at 04:56

It seems that on the validation side (the localhost), the TLS certificate of the remote AWS broker needs to be validated. For that to happen, the validation side, needs to be setup with a trusted keystore(aka truststore) that holds that certificate hierarchy (or the root CA depending on certificate chain returned from the remote AWS broker) from the root CA down to the server certificate.

Please upload to a trusted store the StarField CA certificate and that resolves the issue as per your comment.

This link has details on achieving that.

I ended up using Portecle to download the SSL certificates from the AWS broker and then added those to the client.ts truststore. That resolved the issue. — user928112, Feb 04 '21 at 14:08

Can't connect to broker over SSL

1 Answers1