I had a problem in the process of running kafka and zookeeper on docker compose

Question

this is my docker-compose-single-broker.yml file.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    networks:
      my-network:
        ipv4_address: 172.18.0.100
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 172.18.0.101
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
    networks:
      my-network:
        ipv4_address: 172.18.0.101

networks:
  my-network:
    name: ecommerce-network  # 172.18.0.1 ~

and I executed the command.

docker-compose -f docker-compose-single-broker.yml up -d

I check my network by the command.

docker network inspect ecommerce-network

[
    {
        "Name": "ecommerce-network",
        "Id": "f543bd92e299455454bd1affa993d1a4b7ca2c347d576b24d8f559d0ac7f07c2",
        "Created": "2021-05-23T12:42:01.804785417Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "cad97d79a92ea8e0f24b000c8255c2db1ebc64865fab3d7cda37ff52a8755f14": {
                "Name": "kafka-docker_kafka_1",
                "EndpointID": "4c867d9d5f4d28e608f34247b102f1ff2811a9bbb2f78d30b2f55621e6ac6187",
                "MacAddress": "02:42:ac:12:00:65",
                "IPv4Address": "172.18.0.101/16",
                "IPv6Address": ""
            },
            "f7df5354b9e114a1a849ea9d558d8543ca5cb02800c5189d9f09ee1b95a517d6": {
                "Name": "kafka-docker_zookeeper_1",
                "EndpointID": "b304581db258dd3da95e15fb658cae0e40bd38440c1f845b09936d9b69d4fb23",
                "MacAddress": "02:42:ac:12:00:64",
version: '2'
                "IPv4Address": "172.18.0.100/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

and I entered kafka container. I executed the command to look up the topic list.

enter image description here

however, I couldn't get the topic list even though I waited indefinitely.

this is my kafka container's logs.

enter image description here

What should I do to solve this problem?

score 0 · Answer 1 · answered May 25 '21 at 12:29

Unclear why you think you'll need IP addresses. Remove those

For example, KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 works out of the box with Docker networking

The KAFKA_ADVERTISED_HOST_NAME can simply be localhost if you don't plan on connecting outside of that container, otherwise, it can be the Docker service name you've set of kafka

And you don't need to mount the Docker socket

Related - Connect to Kafka running in Docker

score 0 · Answer 2 · answered May 27 '21 at 03:16

For local development purposes, you could check out my containerized Kafka that uses only single image for both Zookeeper and Kafka if you interested in it

You could find it via https://github.com/howiesynguyen/Java-basedExamples/tree/main/DockerContainerizedKafka4Dev Or https://hub.docker.com/repository/docker/howiesynguyen/kafka4dev

I tested it with one of my examples of Spring Cloud Stream and it worked for me. Hopefully someone can find it helpful

score 0 · Answer 3 · answered Aug 24 '22 at 13:19

Just in case it would be useful to anybody.

I had pretty similar setup as question's author did and in my case Kafka didn't see Zookeeper (but I was using different image - from Debezium). In my case I figured out that environment variable KAFKA_ZOOKEEPER_CONNECT should be named just ZOOKEEPER_CONNECT.

Solution that worked to me (network can be omitted, it's not necessary):

version: "3.9"

services:
  zookeeper:
    image: debezium/zookeeper
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888 
    networks:
      - main
  kafka:
    image: debezium/kafka
    ports:
      - 9092:9092
    environment:
      ZOOKEEPER_CONNECT: zookeeper
    depends_on:
      - zookeeper
    networks:
      - main
networks:
  main: