0

I'm using the Docker compose file below to deploy a simple Kafka server in a container. I also deploy a Nodejs application that bootstraps a consumer on the topic 'my-topic'. So I need the topic to be available when I run the docker compose up command.

I keep getting the "Connection to node" errors in the Kafka container logs indicating that my simple kafka-topics.sh script is failing. Could anyone provide some insight on the issue, or suggest some workarounds?

Compose file:

services:
  nodeapp:
    build: .
    container_name: nodeapp
    depends_on:
      - kafka
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports:
      - '2181:2181'
    networks:
      - kafka
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:3.4.0-debian-11-r21'
    container_name: kafka
    command:
      ["kafka-topics.sh", "--bootstrap-server", "localhost:9092", "--topic", "my-topic", "--create"]
    ports:
      - '9093:9093'
    networks:
      - kafka
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTAINER:PLAINTEXT
      - KAFKA_CFG_LISTENERS=CONTAINER://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=CONTAINER://localhost:9092
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CONTAINER
    depends_on:
      - zookeeper
networks:
  kafka:
    name: kafka-network

Logs:

kafka      | WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
kafka      | [2023-05-03 22:25:30,465] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:30,577] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:30,780] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:30,986] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:31,389] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:32,195] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:33,404] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:34,614] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
kafka      | [2023-05-03 22:25:35,823] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I've tried changing the advertised listeners environment variables but could not get a connection established.

I've confirmed that I can create the topic by using bash on the running Kafka container IF I remove the command that attempts to create the topic from the compose file. This command works for that use case:

docker exec -it /bin/bash kafka
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --create

Please note that the issue above is not the same as this one posted by Sasha Shpota. Sasha's issue is the host machine not being able to resolve the domain. My issue is that the command in the compose file, which should be running in the container, is throwing an error and may be the root cause of the issue.

Note that if I remove the command in the compose file that creates the topic, I'm able to use the docker exec command I mentioned above to create the topic. However, when that line is present in the compose file I get the connection error when executing the same docker exec command.

ngAlchemist
  • 51
  • 5
  • As the error says, broker may not be available... So, is it? At localhost (**within the container**)? 1) Use `docker compose exec` rather than start a brand new container. 2) Changing advertised listeners is correct format node app to communicate with Kafka 3) The Kafka container does not start immediately 4) You should run your topic create script from the node app itself using KafkaJS AdminClient, for example 5) `command` cannot be set on the bitnami container. – OneCricketeer May 04 '23 at 02:11
  • Thanks for suggesting to create the topic in the node app, that's a good idea. I'd still like to figure out the root cause of the issue though. I think the `docker compose exec` command you suggested I try is the same as the `docker exec` command I mentioned above. As I said above, I'm able to execute the Kafka scripts with `docker exec`, so long as I remove the command from the compose file. – ngAlchemist May 04 '23 at 15:59
  • Remove `command:` from the compose file. Then `docker exec` and `docker compose exec` should work. You can find an end-to-end example [here](https://github.com/OneCricketeer/apache-kafka-connect-docker#start-kafka-cluster-in-docker) – OneCricketeer May 05 '23 at 14:57
  • I believe the root cause of this issue is the timing of the compose command. The command is most likely run before the container runs the build scripts for the image. I removed the command from the Dockerfile and was able to create the topic from my node application. – ngAlchemist May 05 '23 at 19:36
  • 1
    Yes, order is important, but also the bitnami container has its own default `command`. Setting your own will override this, and not start the Kafka server correctly – OneCricketeer May 05 '23 at 22:07
  • Thanks @OneCricketeer that explains everything. I hadn't considered that the command would override the bitnami container command. – ngAlchemist May 09 '23 at 18:20

0 Answers0