"kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection" ONLY DURING LISTING TOPICS

Question

I found few questions with similar topic but different context: I can connect to create a Topic but I can't list the topics because I got the error mentioned below (as far as I could see, people were facing issues for basic connecting while I am getting only for listing the topic list).

In case it matters, here is my docker-compose.yml:

version: "3"
services:
    zookeeper:
        image: wurstmeister/zookeeper

    kafka:
        image: wurstmeister/kafka
        ports:
            - "9092:9092"
        environment:
            KAFKA_ADVERTISED_HOST_NAME: "localhost"
            KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"

My console:

bash-4.4# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test2
Created topic test2.
bash-4.4# kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
        at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

bash-4.4# kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test3
Created topic test3.
bash-4.4# kafka-topics.sh --list --zookeeper localhost:2181
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
        at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:321)
        at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
        at kafka.admin.TopicCommand.main(TopicCommand.scala)

Edited

Future readers may be found useful how I could list all topics straight from my Docker Kafka Container without logging in my Docker Zookeper Container (https://stackoverflow.com/a/56595227/4148175)

C:\Users\>docker exec -it test1_kafka_1 bash

bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
test2
test3
test_topic
bash-4.4#

OneCricketeer · Accepted Answer · 2021-11-30T16:10:06.703

25

--zookeeper zookeeper:2181 seems to have worked fine

--zookeeper localhost:2181 will always fail inside the kafka container because it's not running a zookeeper server

I could list all topics straight from my Docker Kafka Container without logging in my Docker Zookeper Container

That's right. Ideally, you should never enter the Zookeeper container. Latest kafka versions support using --bootstrap-server instead, so you could use kafka:9092 or localhost:9092 from the kafka container

edited Nov 30 '21 at 16:10

answered Jan 23 '20 at 05:50

OneCricketeer

179,855
19
132
245

You have certainly answered my root issue. Kindly, could you add here how would you list all topics when you have zookeper in one docker container and kafka in another? If you note my docker-compose I started two diferent containers. – Jim C Jan 23 '20 at 13:10
1

I feel like I already answered that. Docker isn't the problem. It's a lack of providing the correct server address in your commands. Localhost will always resolve to the current machine/container, and Zookeeper is always recommended not to run on the same machine as the broker in a real kafka deployment – OneCricketeer Jan 23 '20 at 15:04
Thanks a lot. A last question if you don't mind: what was my question downvoted? I have really search around and I didn't find anyone getting the same error trying to list the topics (well, now I know what it wasn't working but it doesn't seem I tried something worthless or plenty answered around) – Jim C Jan 23 '20 at 18:23
I'm not sure. You cannot see who downvoted. Most likely because you have a typo – OneCricketeer Jan 23 '20 at 18:25
1

BTW, I finally found what I was looking for. I will add above for future readers. But certainly your answer which deals with my issue was important so I could narrow my searches around. – Jim C Jan 23 '20 at 18:32
In my case "--zookeeper zookeeper:2181" do not work. Instead, I solved changing the command with this: "kafka-topics --create --topic test --partitions 1 --replication-factor 1 --if-not-exists --bootstrap-server localhost:9092" – DAme Nov 30 '21 at 16:01
1

@DAme Correct, as mentioned in my answer, `--zookeeper` flag is being deprecatd – OneCricketeer Nov 30 '21 at 16:08

"kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection" ONLY DURING LISTING TOPICS

Edited

1 Answers1

Linked

Related