Bitnami Kafka KRaft Failed to configure SaslClientAuthenticator

Question

I'm trying to migrate Kafka with ZooKeeper to KRaft and I have a problem with inter-broker secured setup. Everithing works nicely with disabled authorizer via CONTROLLER with PLAINTEXT, but when I'm trying to setup it, Kafka throws errors below. Does someone have an idea, what's wrong with it?

TY Here is my Kafka setup:

docker-compose.yml:

version: "2"

services:
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - 9092:9092
      - 9093:9093
    restart: always
    volumes:
      - "kafka_data:/bitnami"
      - ./certs/keystore.p12:/bitnami/kafka/config/certs/kafka.keystore.jks
      - ./certs/truststore.jks:/bitnami/kafka/config/certs/kafka.truststore.jks
      - ./config/server.properties:/bitnami/kafka/config/server.properties
      - ./config/kafka_jaas.conf:/opt/bitnami/kafka/config/kafka_jaas.conf
      - ./config/admin.properties:/opt/bitnami/kafka/config/admin.properties
    environment:
      - BITNAMI_DEBUG=true
      - ALLOW_PLAINTEXT_LISTENER=no
      - KAFKA_CERTIFICATE_PASSWORD=123
      - KAFKA_TLS_TYPE=JKS
      - KAFKA_KRAFT_CLUSTER_ID=Hd3vEcAzTR3Flf0Ig
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=SASL_SSL://:9092,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,SASL_SSL:SASL_SSL
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=SASL_SSL
      - KAFKA_CFG_ADVERTISED_LISTENERS=SASL_SSL://kafka:9092
      - KAFKA_CLIENT_USERS=admin
      - KAFKA_CLIENT_PASSWORDS=123
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka:9093

volumes:
  kafka_data:
    driver: local

server.properties:

node.id=1
controller.quorum.voters=1@kafka:9093
listeners=SASL_SSL://:9092,CONTROLLER://:9093
inter.broker.listener.name=SASL_SSL
advertised.listeners=SASL_SSL://kafka:9092

controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/bitnami/kafka/data
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=24
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

group.initial.rebalance.delay.ms=0
delete.topic.enable=true
auto.create.topics.enable=true

advertised.host.name=kafka

security.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
#security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=PLAIN

listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="123" \
  user_admin="123";
 
super.users=User:admin

ssl.keystore.location=/bitnami/kafka/config/certs/kafka.keystore.jks
ssl.keystore.password=123
ssl.key.password=123
ssl.truststore.location=/bitnami/kafka/config/certs/kafka.truststore.jks
ssl.truststore.password=23

ssl.endpoint.identification.algorithm=
ssl.client.auth=required

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=false

kafka_jaas.conf:

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  serviceName="kafka"
  username="admin"
  password="123"
  user_admin="123";
};

Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="123";
};

Logs, that Kafka throws:

29.03.2023 15:11:51
[2023-03-29 12:11:51,234] INFO [BrokerToControllerChannelManager broker=1 name=heartbeat]: Recorded new controller, from now on will use node kafka:9093 (id: 1 rack: null) (kafka.server.BrokerToControllerRequestThread)
29.03.2023 15:11:51
[2023-03-29 12:11:51,235] WARN [BrokerToControllerChannelManager broker=1 name=heartbeat] Error connecting to node kafka:9093 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
29.03.2023 15:11:51
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
29.03.2023 15:11:51
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:992)
29.03.2023 15:11:51
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1(InterBrokerSendThread.scala:103)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1$adapted(InterBrokerSendThread.scala:99)
29.03.2023 15:11:51
    at scala.collection.Iterator.foreach(Iterator.scala:943)
29.03.2023 15:11:51
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
29.03.2023 15:11:51
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
29.03.2023 15:11:51
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
29.03.2023 15:11:51
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
29.03.2023 15:11:51
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.sendRequests(InterBrokerSendThread.scala:99)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.pollOnce(InterBrokerSendThread.scala:73)
29.03.2023 15:11:51
    at kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManager.scala:421)
29.03.2023 15:11:51
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
29.03.2023 15:11:51
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
29.03.2023 15:11:51
    at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:239)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
29.03.2023 15:11:51
    ... 16 more
29.03.2023 15:11:51
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
29.03.2023 15:11:51
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
29.03.2023 15:11:51
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:632)
29.03.2023 15:11:51
    at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:202)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:285)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:228)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:143)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:236)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
29.03.2023 15:11:51
    at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
29.03.2023 15:11:51
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:992)
29.03.2023 15:11:51
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1(InterBrokerSendThread.scala:103)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.$anonfun$sendRequests$1$adapted(InterBrokerSendThread.scala:99)
29.03.2023 15:11:51
    at scala.collection.Iterator.foreach(Iterator.scala:943)
29.03.2023 15:11:51
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
29.03.2023 15:11:51
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
29.03.2023 15:11:51
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
29.03.2023 15:11:51
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
29.03.2023 15:11:51
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.sendRequests(InterBrokerSendThread.scala:99)
29.03.2023 15:11:51
    at kafka.common.InterBrokerSendThread.pollOnce(InterBrokerSendThread.scala:73)
29.03.2023 15:11:51
    at kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManager.scala:421)
29.03.2023 15:11:51
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)

BTW I have no any Kerbetos settingsin my setup. But you can see this error.

Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login

I've tryed to use multiple variations with settings with no result...

atisu · Answer 1 · 2023-07-25T12:47:58.977

I had the same issue, and I was able to fix it by including the following in the server.properties file (or add it to the Docker environment):

sasl.mechanism.controller.protocol=PLAIN

Based on the documentation [1] the default value is GSSAPI (Kerberos). You can remove the serviceName from the JAAS config.

[1] https://docs.confluent.io/platform/current/installation/configuration/broker-configs.html#sasl-mechanism-controller-protocol

Bitnami Kafka KRaft Failed to configure SaslClientAuthenticator

1 Answers1