kafka container - Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available

Question

I'm following this guide in order to start a single kafka container .I'm working on win10 pro and my cli is gitbash.

docker-compose-single-broker.yml :

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka 
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The containers are starting without any problem :

$ docker-compose -f docker-compose-single-broker.yml up -d
Creating kafka-docker_zookeeper_1 ... done
Creating kafka-docker_kafka_1     ... done

$ docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
19929c6c3297        wurstmeister/kafka       "start-kafka.sh"         17 seconds ago      Up 15 seconds       0.0.0.0:9092->9092/tcp                               kafka-docker_kafka_1
d343a8ecf7ed        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   17 seconds ago      Up 15 seconds       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka-docker_zookeeper_1

However, when I try to run any kafka command(create topic,list topics) from inside the container like the following :

> $KAFKA_HOME/bin/kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`
> $KAFKA_HOME/bin/kafka-topics.sh --describe --topic test --bootstrap-server `broker-list.sh`

I'm getting the following warnings and afterwards a timeout exception :

$ ./start-kafka-shell.sh localhost
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic test --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`
[2020-06-09 11:47:26,241] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/172.17.54.145:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-06-09 11:47:29,306] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/172.17.54.145:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

The content of the start-kafka-shell.sh :

$ cat start-kafka-shell.sh
#!/bin/bash
docker run --rm -v //var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash

Using your exact Docker Compose and running the commands in the Kafka container as described works just fine for me. Are you definitely running these commands from within the Kafka broker machine? — Robin Moffatt, Jun 09 '20 at 11:42
Any chance u arent working on windows ? or using a different docker version ? — JeyJ, Jun 09 '20 at 11:45
I am working on a Mac. Please can you update your question to show how you are running the commands in docker, e.g. `docker exec` — Robin Moffatt, Jun 09 '20 at 11:47

score 0 · Answer 1 · answered Jun 09 '20 at 16:33

0

I'm not sure what the purpose of start-kafka-shell.sh is. If you're using Docker then you can interact with it from your Windows shell as follows:

docker exec -it tmp_kafka_1 bash -c '$KAFKA_HOME/bin/kafka-topics.sh --create --topic test4 --partitions 1 --replication-factor 1 --bootstrap-server `broker-list.sh`'


docker exec -it tmp_kafka_1 bash -c '$KAFKA_HOME/bin/kafka-topics.sh --describe --topic test4 --bootstrap-server `broker-list.sh`'

Topic: test4    PartitionCount: 1       ReplicationFactor: 1    Configs: segment.bytes=1073741824
        Topic: test4    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

You just need the correct Docker container name (tmp_kafka_1 in my example above), which you can confirm with docker ps.

answered Jun 09 '20 at 16:33

Robin Moffatt

30,382
3
65
92

The start-kafka-shell.sh just attaches my cli to to the container`s shell. I added to the post its content.When I run what u suggested I'm getting the same output . Maybe related to the fact that I'm working on windows ? – JeyJ Jun 09 '20 at 19:44
What about if you run `docker exec -it tmp_kafka_1 bash -c '$KAFKA_HOME/bin/kafka-topics.sh --create --topic test4 --partitions 1 --replication-factor 1 --bootstrap-server localhost:9092'` ? – Robin Moffatt Jun 09 '20 at 19:47
exactly the same :( I can reproduce the error whenever I want (from scratch) if that is usefull.. – JeyJ Jun 09 '20 at 19:50
An important note, when I try to produce and consume from localhost:9200 I succeed (tried with java libraries..) – JeyJ Jun 09 '20 at 20:10
I'd be inclined to try this then: https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html – Robin Moffatt Jun 09 '20 at 21:47
The start script is in the wurstmeister repo – OneCricketeer Jun 17 '20 at 17:13

kafka container - Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available

1 Answers1