Kafka docker and LEADER_NOT_AVAILABLE

Question

I use the following Kafka Docker image: https://hub.docker.com/r/wurstmeister/kafka/

I'm able to start Apache Kafka with the following properties:

<KAFKA_ADVERTISED_HOST_NAME>${local.ip}</KAFKA_ADVERTISED_HOST_NAME>
<KAFKA_ADVERTISED_PORT>${kafka.port}/KAFKA_ADVERTISED_PORT>     
<KAFKA_ZOOKEEPER_CONNECT>zookeeper:2181</KAFKA_ZOOKEEPER_CONNECT>   
<KAFKA_MESSAGE_MAX_BYTES>15000000</KAFKA_MESSAGE_MAX_BYTES>

but I see the following warning when trying to send the message into the topic:

WARN 9248 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {post.sent=LEADER_NOT_AVAILABLE}

I saw a few articles on the internet that told that this issue can be related to old properties like KAFKA_ADVERTISED_HOST_NAME and KAFKA_ADVERTISED_PORT and I should reconfigure to KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS. But when I start the Kafka container with the following properties:

<KAFKA_ADVERTISED_LISTENERS>PLAINTEXT://${local.ip}:${kafka.port}</KAFKA_ADVERTISED_LISTENERS>
<KAFKA_LISTENERS>PLAINTEXT://${local.ip}:${kafka.port}</KAFKA_LISTENERS>                            
<KAFKA_ZOOKEEPER_CONNECT>zookeeper:2181</KAFKA_ZOOKEEPER_CONNECT>   
<KAFKA_MESSAGE_MAX_BYTES>15000000</KAFKA_MESSAGE_MAX_BYTES>

my application unable to connect to Kafka:

2018-08-25 16:20:57.407  INFO 17440 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version : 1.1.0
2018-08-25 16:20:57.408  INFO 17440 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId : fdcf75ea326b8e07
2018-08-25 16:20:58.513  WARN 17440 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 could not be established. Broker may not be available.
2018-08-25 16:20:59.567  WARN 17440 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient   : [AdminClient clientId=adminclient-1] Connection to node -1 could not be established. Broker may not be available.

How to properly reconfigure the Docker Kafka in order to be able to use KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS?

Ah, and `local.ip` is being assigned as what by Maven? Because I know if you set the container hostname as kafka, for example, and use `PLAINTEXT://kafka:9092` then it'll at least connect externally after you forward a port — OneCricketeer, Aug 25 '18 at 16:59
See if my answer here helps https://stackoverflow.com/a/51634499/2308683 — OneCricketeer, Aug 25 '18 at 17:00

score 0 · Answer 1 · answered May 29 '21 at 13:51

From this awesome post, here's a good explanation about these properties:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.

When your application connects to one of the addresses from LISTENERS the Kafka returns the correspondent KAFKA_ADVERTISED_LISTENER to that LISTENER you choosed. The KAFKA_ADVERTISED_LISTENER returned is the address that your application you will really use to communicate with Kafka.

So, you have to use in your application what you set on LISTENERS Kafka property for PLAINTEXT.

Using this configuration as you put:

<KAFKA_ADVERTISED_LISTENERS>PLAINTEXT://${local.ip}:${kafka.port}</KAFKA_ADVERTISED_LISTENERS>
<KAFKA_LISTENERS>PLAINTEXT://${local.ip}:${kafka.port}</KAFKA_LISTENERS>

You have to use on your application:

As you used on Kafka docker ${local.ip}:${kafka.port} you have to get the assigned kafka docker container IP and use it in your application.

Just to fill the variables for this scenario, let's suppose your kafka docker container IP 192.250.0.1 and the kafka port used is 9092, so your application bootstrap.servers property would be: 192.250.0.1:9092

Here's a command to see what Kafka returns to you when you try to connect using one of kafka's listener:

$ kafkacat -b 192.250.0.1:9092 -L

kafkacat is a very useful tool for test and debug kafka.

Kafka docker and LEADER_NOT_AVAILABLE

1 Answers1