Failed SSL handshake in Kafka Producer

Question

I'm trying to set up a Kafka broker with SSL enabled communication. I followed the guide as described in the Kafka documentation here under 7.2: https://kafka.apache.org/documentation/#security

When I'm trying to execute the command ./kafka-console-producer.sh --broker-list ec2-ip.eu-central-1.compute.amazonaws.com:9093 -topic test --producer.config ../config/producer.properties I get:

ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-08-22 13:15:56,728] ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

Testing the SSl connection in general with

openssl s_client -debug -connect ec2-ip.eu-central-1.compute.amazonaws.com:9093 -tls1

results in

0020 - a0 41 45 81 42 b6 83 d3-2b 94 02 9f ac 42 73 42   .AE.B...+....BsB
---
Certificate chain
 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
   i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDuTCCAqGgAwIBAgIEPsshdTANBgkqhkiG9w0BAQsFADBsMRAwDgYDVQQGEwdV
bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD
VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMRAwDgYDVQQDEwdVbmtub3du
MB4XDTE5MDgyMjEyNTIwOFoXDTIwMDgyMTEyNTIwOFowbDEQMA4GA1UEBhMHVW5r
bm93bjEQMA4GA1UECBMHVW5rbm93bjEQMA4GA1UEBxMHVW5rbm93bjEQMA4GA1UE
ChMHVW5rbm93bjEQMA4GA1UECxMHVW5rbm93bjEQMA4GA1UEAxMHVW5rbm93bjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAI5EKxmSDLQvW/ePu9mB3qdf
E4cG1ok1kI48S6yL4gchx2egIC6Zjr5gBhuek36mhKV82H1nVYfDzJVyxoyaTZ5q
ux8xq+ooNN/chKHy0ExpCZ2HHvrQiTPQit//dyA5zc5yOyRcJo/m2EhYfRMuLfFM
kx1mEg2cs6Qa/hd6jKjCSvbcKDfrOhf4rLagHzI+2GRQfF6LXxYBE3Ti7u6fkqog
o67CTrNR6IER1b4wGIeHga8eXiVhOuP4JiWdpF69i5py0RVfzsesP0W8RB2qElte
+7EwShaFO72Xa7MPiDz1rE4D5pohQ09AWGb/B0YIcI7qh9jyInbj4eWhhlnd1lsC
AwEAAaNjMGEwQAYDVR0RBDkwN4I1ZWMyLTM1LTE1OC0yMzItMTc2LmV1LWNlbnRy
YWwtMS5jb21wdXRlLmFtYXpvbmF3cy5jb20wHQYDVR0OBBYEFNGugrNUmFiVKVyU
Uiepd0OUme1wMA0GCSqGSIb3DQEBCwUAA4IBAQAToYYUsvQ6MlOQ+HHWnfax5L0u
M/IrExjcrSmnxAevnTem5+Hnz+AmlRHSF1/FfCWhi2XMSHBHXU52uhIt+kDLNhFh
D17Je0FydthW8roBsLZ+XsuHjBi8MXo41YAA19+cjW60/se3+kWxOHH1rEfV9ky/
K9kWufQPl3U3fbUzh0rWwriSDaPUTjHg/IeROhVe8gt/QeBfTU2KUqnln6K58PfC
kE2c2zCTQwpsfwi5gqvrdqWZc/A5qriv1l39zKU1icfTZzdeejcjtbCdFqajOqar
lMXwyl1bPuPiOHas7K9J9MSREQYql89gbN5V9gO//IHmM97eLt4npFSpu8Cp
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1451 bytes and written 236 bytes
Verification error: self signed certificate
---
New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 5D5E9882C4E8DD80427108B85C7649F0EC1AA87B9DAA0EA1FCE8C7063C83A61B
    Session-ID-ctx: 
    Master-Key: 861677EDA9E19E3D8926889A9B0DE299593C7FCD49DB8A55EBF4D222800169E16CDB74DCE0EC392A3B491268FCCF5F07
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1566480514
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: yes
---

which looks ok to me.

Configuration of the Broker:

listeners=SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093,PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092
group.initial.rebalance.delay.ms=0
advertised.listeners=PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092,SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093
ssl.endpoint.identification.algorithm=SSL

ssl.keystore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.keystore.jks
ssl.keystore.password=Welcome01
ssl.key.password=Welcome01
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.truststore.jks
ssl.truststore.password=Welcome01
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS

Configuration of the Producer:

bootstrap.servers=localhost:9092
security.protocol=SSL
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/client.truststore.jks
ssl.truststore.password=Welcome01
logging.level.org.apache.kafka: DEBUG
log4j.rootLogger=DEBUG

Do you see some misconfiguration? How can I proceed to investigate further to get down to the issue?

Thanks to mazaneiche I was able to get the error that leads to the failed handshake:

kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown

Why does this error occur? How can it be fixed?

Any reason why using -tls1 in open ssl test , tried testing it without mentioning the protocol version i.e the Default one — Soumen Mukherjee, Aug 22 '19 at 15:25
one way to find out is to use `$ export KAFKA_OPTS="-Djavax.net.debug=ssl:handshake:verbose"` before running your producer. — mazaneicha, Aug 22 '19 at 22:16
@mazaneicha Thanks a lot. That helped as I do now have a proper error. But still I don't know what the problem is/what the cause of the error is. Can you help? I edited my initial question and added the error from the verbose handshake. — RenPen, Aug 23 '19 at 09:40
`unable to find valid certification path to requested target` -- you have a problem with the truststore used by producer. I'd validate that you completed all the steps from your tutorial, or follow this guide https://docs.confluent.io/2.0.0/kafka/ssl.html. — mazaneicha, Aug 23 '19 at 12:37
@mazaneicha Thanks for the link. I did the tutorial again. Now I get another error: `java.security.cert.CertificateException: No name matching ec2-ip.eu-central-1.compute.amazonaws.com found %% Invalidated: [Session-4, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256] kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown kafka-producer-network-thread | console-producer, WRITE: TLSv1.2 Alert, length = 2 kafka-producer-network-thread | console-producer, fatal: engine already closed.` Do you know what is wrong now? — RenPen, Aug 26 '19 at 17:53
This usually means that the host name doesn't match the CN in a cert that it sent. See https://stackoverflow.com/questions/3093112/certificateexception-no-name-matching-ssl-someurl-de-found. As last resort, you can try disabling hostname verification by setting another java option for producer `-Dssl.endpoint.identification.algorithm= ` (<- this is empty string). — mazaneicha, Aug 27 '19 at 13:31
Also, possible values for `ssl.endpoint.identification.algorithm` are HTTPS, LDAPS and blank (none) -- https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html. Your config uses 'SSL' which I'm not sure how Kafka will handle. — mazaneicha, Aug 27 '19 at 13:46
I tried it with a blank "ssl.endpoint.identification.algorithm=" just to see if it works then but I still get the same error. Also tried =HTTPS. Still the same error :( Any other ideas? — RenPen, Sep 01 '19 at 11:21

score 0 · Answer 1 · answered Jul 13 '21 at 14:40

Apparently not the issue here but I have been struggling with the same error for a while and in my case the kafka-console-producer.sh was failing with the similar errors:

$ kafka-console-producer.sh --broker-list 192.168.1.173:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

checking with debug

$ export KAFKA_OPTS="-Djavax.net.debug=ssl"

kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
java.security.cert.CertificateException: No subject alternative names present
%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
[2021-07-13 16:21:53,909] ERROR [Producer clientId=console-producer] Connection to node -1 (/192.168.1.173:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

In my case turned out the reason is using IP instead of hostname when specifying broker-list parameter.

That worked:

$ kafka-console-producer.sh --broker-list red03:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties

Failed SSL handshake in Kafka Producer

1 Answers1