0

Faced with the problem of authentication in the kafka topic using SSL from spark-streaming.

I've got 3 ssl certs in pem format for authentication in the kafka topic:

  1. ssl_cafile
  2. ssl_certfile
  3. ssl_keyfile.

In kafka-python I'm using them in such way:

producer = KafkaProducer(bootstrap_servers=hostname,
                         security_protocol=security_protocol,
                         ssl_check_hostname=False,
                         ssl_cafile='./certs_kafka/CARootChain.pem', 
                         ssl_certfile='./certs_kafka/certificate.pem',
                         ssl_keyfile='./certs_kafka/key.pem')

How can I use them in spark-streaming to auth my topic?

Looking up in google for the answer I've found such example for ssl auth:

df = spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", "server:port") \
.option("subscribe", "z-main-like-test-topic") \
.option("startingOffsets", "earliest") \
.option("failOnDataLoss", "false") \
.option("kafka.security.protocol", "SSL") \
.option("kafka.ssl.enabled.protocols", "TLSv1.2") \
.option("kafka.ssl.truststore.location", truststore) \
.option("kafka.ssl.truststore.type", "JKS") \
.option("kafka.ssl.truststore.password", truststore_password) \
.option("kafka.ssl.keystore.location", keystore) \
.option("kafka.ssl.keystore.type", "JKS") \
.option("kafka.ssl.keystore.password", keystore_password) \
.load()

But I really don't understand what should be put in the keystore and what in the trustore. Maybe someone faced with ssl auth in spark-streaming and can explain which pem to combine with which to make a keystore/trustore jks...

nagibator_archivator
  • 1
  • 1
  • See this? https://stackoverflow.com/questions/22296312/how-to-convert-certificate-from-pem-to-jks#22298627 – OneCricketeer Apr 21 '23 at 13:39
  • @OneCricketeer and what should be placed in truststore then? – nagibator_archivator Apr 21 '23 at 18:02
  • I'm not completely sure. There are other questions on asking about [PEM files and TrustStores](https://stackoverflow.com/questions/25352081/ca-pem-file-to-jks-tomcat-truststore). All of them are using `keytool` and `openssl` command to convert files – OneCricketeer Apr 21 '23 at 19:55
  • In any case, Kafka also accepts PEM files. Therefore, so does SparkSQL's Kafka integration https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key – OneCricketeer Apr 21 '23 at 19:56

0 Answers0