How to start a dockerized Kafka instance and connect to it from the host machine?

Question

How do I do the following:

Start a dockerized instance of Kafka on host machine1
Run a process on machine1 which can consume or produce Kafka messages

Here is what I have so far:

I installed the following items on my host machine. (machine1):
docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
This is a Linux computer running Debian 11. It exists on my local network. I am connected to it from a Windows computer via ssh.
I added my user to the group docker.
I downloaded the following YAML file to obtain a docker-compose.yml file:
curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml
That looks like this:

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

I don't know much about Docker at all, and my experience with it is limited, however this looks ok. I would like to use the bitnami containers, as I have worked with these before, albeit in a limited capacity.

I note that the port numbers look correct.

Finally, I started the docker containers with docker compose up -d
I can see the containers started with docker ps

$ docker ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                                     NAMES
b56084cdd443   bitnami/kafka:3.4       "/opt/bitnami/script…"   3 seconds ago   Up 2 seconds   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                 kafka-kafka-1
0e6b30a47d06   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   4 seconds ago   Up 3 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   kafka-zookeeper-1

What works:

I found that I was able to create a Kafka Topic by executing a bash session inside of the Kafka container. I was also able to Produce and Consume messages using the helper utilities.

The three utilities of interest live here:

$ docker exec -it kafka-kafka-1 bash
I have no name!@b56084cdd443:$ cd /opt/bitnami/kafka/bin
I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ls
...
kafka-console-consumer.sh
kafka-console-producer.sh
kafka-topics.sh

They appear to be working. For example, I created test-topic with this command:

I have no name!@b56084cdd443:/opt/bitnami/kafka/bin$ ./kafka-topics.sh 
 \
--bootstrap-server localhost:9092 --create --topic test-topic

What does not work:

I was not able to Consume or Produce messages using kafkacat

This was my next stage in testing.

I installed kafkacat on the host machine (machine1):
sudo apt install kafkacat
I ran kafkacat to inspect the topics: kafkacat -L -t test-topic -p localhost:9092

$ kafkacat -L -t test-topic -b localhost:9092
Metadata for test-topic (from broker -1: localhost:9092/bootstrap):
 1 brokers:
  broker 1001 at b56084cdd443:9092 (controller)
 1 topics:
  topic "test-topic" with 1 partitions:
    partition 0, leader 1001, replicas: 1001, isrs: 1001

That seems to be ok.

I then became stuck trying to Produce messages and send them to Kafka.

$ kafkacat -P -b localhost:9092 -t test-topic
Hello World
%3|1680812310.642|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: 
b56084cdd443:9092/1001: Failed to resolve 'b56084cdd443:9092': 
Name or service not known (after 36ms in state CONNECT)
%3|1680812311.658|FAIL|rdkafka#producer-1| [thrd:b56084cdd443:9092/1001]: b56084cdd443:9092/1001: 
Failed to resolve 'b56084cdd443:9092': 
Name or service not known (after 15ms in state CONNECT, 1 identical error(s) suppressed)
^\Quit

At this point I reach a blocker and don't know how to proceed.

While trying to research the problem, I found an article which initially looked useful, but wasn't. I didn't understand it, probably because I don't have sufficient background information.

https://www.baeldung.com/kafka-docker-connection

If this article indeed does explain what is wrong, and how to fix it, then could someone explain it to me in further detail so that I can understand what to do and why this doesn't work in its current configuration?

If it turns out not to be relevant then please ignore it.

The README of the Bitnami kafka container page explains how the listeners settings can be used... — OneCricketeer, Apr 07 '23 at 02:26
@OneCricketeer Where? https://docs.bitnami.com/general/infrastructure/kafka/ — FreelanceConsultant, Apr 07 '23 at 07:16
On Github or Docker hub https://github.com/bitnami/containers/tree/main/bitnami/kafka#accessing-apache-kafka-with-internal-and-external-clients — OneCricketeer, Apr 07 '23 at 15:27

score 0 · Accepted Answer · answered Apr 06 '23 at 20:47

I found a solution, which required some editing of my docker-compose.yml file.

$ cat docker-compose.yml
version: "2"

services:
  zookeeper:
    container_name: zookeeper1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    container_name: kafka1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/kafka:3.4
    ports:
      #- "9092:9092"
      - 29092:29092
    volumes:
      - "kafka_data:/bitnami"
    environment:
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
      ALLOW_PLAINTEXT_LISTENER: yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

networks:
  kafka_net:
    name: kafka_net

We can see this now works as expected:

kafkacat -L -t test-topic -b localhost:29092
Metadata for test-topic (from broker 1001: localhost:29092/1001):
 1 brokers:
  broker 1001 at localhost:29092 (controller)
 1 topics:
  topic "test-topic" with 1 partitions:
    partition 0, leader 1001, replicas: 1001, isrs: 1001

And more importantly...

$ kafkacat -P -b localhost:29092 -t test-topic
hello world
^\Quit

$ kafkacat -C -b localhost:29092 -t test-topic
Hello World
These are two test messages.
hello world
hello world
% Reached end of topic test-topic [0] at offset 4

Note that "hello world" is shown twice. One entry was from testing the dockerized kafka-console-producer.sh, the second is from kafkacat.

Also useful: https://www.baeldung.com/kafka-docker-connection https://www.confluent.io/blog/kafka-listeners-explained/ — FreelanceConsultant, Apr 09 '23 at 11:23

How to start a dockerized Kafka instance and connect to it from the host machine?

How do I do the following:

Here is what I have so far:

What works:

What does not work:

1 Answers1