2

I am trying to setup a Kafka cluster with mTLS authentication using certificates signed by GCP's CAS (Certificate Authority Service). I have three Kafka nodes: a master and two workers. Each node has a PEM truststore containing the CA Root certificate from the authority on CAS and a PEM keystore containing a signed certificate from CAS and the private key. I followed this webpage for the setup.

This is the server.properties file for the master node. Other nodes have a similar config except the ssl.keystore.location property.

listeners=INTERNAL://:port,EXTERNAL://:port
advertised.listeners=INTERNAL://:port,EXTERNAL://:port
listener.security.protocol.map=INTERNAL:SSL,EXTERNAL:SSL
inter.broker.listener.name=INTERNAL
ssl.enabled.protocols=TLSv1.2
ssl.endpoint.identification.algorithm=
producer.ssl.endpoint.identification.algorithm=
consumer.ssl.endpoint.identification.algorithm=
ssl.client.auth=required
ssl.truststore.type=PEM
ssl.truststore.location=/path/ca.crt
ssl.keystore.type=PEM
ssl.key.password=<password>
ssl.keystore.location=/path/master.pem

The truststore is the ca.crt (Root CA certificate) file and the keystore has the private key and the signed certificate. There are no intermediate certificates, the certificates in the keystore are directly signed by the root certificate.

When I try to start the server on any nodes I am getting the following error. I am not sure why.

ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.config.ConfigException: Invalid value javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found. for configuration A client SSLEngine created with the provided settings can't connect to a server SSLEngine created with those settings.

P.S. I have already tried using the truststore and keystore in JKS and PKCS12 format. I keep getting the same error.

dippatel
  • 21
  • 3

3 Answers3

1

From what you shared, it doesn't seem that you've created valid keystores for the cert and CA.

ssl.truststore.location=/path/ca.crt
...
ssl.keystore.location=/path/master.pem

You're merely pointing to the original files. You need to create two valid java keystores with a JDK's keytool tool and import the files into each one. Depending on your certificate format you might need to convert them with openssl.

See https://stackoverflow.com/a/11954816/50114 for an example.

Gilles Philippart
  • 637
  • 5
  • 12
1

You mentioned earlier that you got the same error with JKS keystore and truststore. From the error message, it sounds like the keystore may not contain the whole chain. Can you first try with a JKS truststore containing the root cert and a JKS keystore containing the private key and the whole certificate chain (i.e. including root cert)?

If that run gives the same error, you could enable "javax.net.debug=ssl". If using Kafka scripts to start the broker, you can first export KAFKA_OPTS=javax.net.debug=ssl. Broker performs extra validation by attempting handshake using the provided keystore and truststore for the inter-broker listener. It looks like that validation is failing in your case. The extra debug may help to understand why.

Once you have the broker running with JKS, you can switch to PEM if required.

Rajini
  • 11
  • 1
0

What does your keystore PEM file look like?

  1. Does it have clear separate sections (with BEGIN and END)? See https://docs.progress.com/en-US/bundle/datadirect-hybrid-data-pipeline-installation-46/page/PEM-file-format.html
  2. Can you try adding the Root CA in the PEM file too, after the signed certificate?
Gilles Philippart
  • 637
  • 5
  • 12
  • My keystore PEM looks exactly like you mentioned, I don't have any intermediate certificates. -----BEGIN ENCRYPTED PRIVATE KEY----- # PRIVATE KEY HERE -----END ENCRYPTED PRIVATE KEY----- -----BEGIN CERTIFICATE----- # PUBLIC KEY HERE -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- # ROOT CERTIFICATE HERE -----END CERTIFICATE----- – dippatel Nov 23 '22 at 16:16
  • Is your file a one-liner or do you have line breaks? – Gilles Philippart Nov 24 '22 at 10:54
  • It does have line breaks, I added them to the comment but stackoverflow removed them. – dippatel Nov 28 '22 at 15:12