0

I am working on leveraging confluent kafka s3 connector to download kafka record and save them as parquet file in minio. It worked fine with a dummy kafka without authentication.

Now I'm trying to verify the flow with a real kafka instance that requires kerberos authentication.

I assured my kerberos credentials are in place and set up properly.

env | grep OPTS
KAFKA_OPTS=-Djava.security.krb5.conf=/plugins/krb5.conf -Djava.security.auth.login.config=/plugins/kafka_client_jaas.conf

in my connector.properties file I specified the following

#connecting to kafka
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

Now I start the connector-standalone.sh

root@2c553a4e0b7c:/opt/bitnami/kafka/bin# ./connect-standalone.sh /plugins/connector.properties /plugins/s3-sink.properties

[2022-09-21 10:37:43,559] WARN [Consumer clientId=connector-consumer-s3-sink-0, groupId=connect-s3-sink] Bootstrap broker broker:9030 (id: -1 rack: n
ull) disconnected (org.apache.kafka.clients.NetworkClient:1024)

It seems that the krb5 authentication is not enabled. I've tried kafka-console-consumer.sh with the same krb configs and credentials and it all worked ok.

It's likely that the three lines of configuration in connector.properties did not take effective. Are the correct configurations to to notify worker to use krb5?

security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
Jin Ma
  • 169
  • 2
  • 12
  • I verified the keys and krb5 config files. they are all ok as using kafka-console-consumer.sh I can successfully consume messages from kafka. – Jin Ma Sep 23 '22 at 16:17

1 Answers1

0

Kafka-connect, Bootstrap broker disconnected

Answers to this question apply to here as well.

There's no document or whatsoever that talks about CONNECT_CONSUMER env vars must be set and repeat the jvm parameters.

Jin Ma
  • 169
  • 2
  • 12