I have written two test applications which interface to Kafka. One of those applications is containerized inside a Docker container. Kafka is also containerized, running inside a Docker container. The specific Kafka image used is the Bitnami one.
The other application runs on the same machine (a physical Linux machine) which hosts the Docker containers.
Here is a description of the two applications:
- A Java application which runs on the docker host. (Same machine as the machine for which
docker ps
, etc, displays information about currently running containers.) This application was able to network with the containerized Kafka container via port 29092. This is the only port which is declared in thedocker-compose.yml
file for the Kafka container.
2: A Rust applicaiton which runs in its own Dockerized development environment. It is "adjacent" to the Dockerized Kafka instance, and both containers are connected to the same Docker network. (Or at least they should be.) This application does not work and cannot connect to Kafka via port 29092. I do not understand why. However, it is able to connect to Kafka via port 9092. Again, I do not understand why this is the case.
Here are some further details of the configuration:
Kafka docker-compose.yml
I don't fully understand what EXTERNAL_SAME_HOST
and INTERNAL
is doing in this configuration. What I do understand is that these are some kind of environment variable which the Kafka process/application picks up when it first starts and causes a socket to start listening to the specified address range and ports. Quite what the difference between "EXTERNAL_SAME_HOST" and "INTERNAL" is I am not sure. These strings seem to vary quite a bit depending on different examples, so I am not sure if they are arbitrary or what the possible values are.
version: "2"
services:
zookeeper:
container_name: zookeeper1
networks:
- "kafka_net"
image: docker.io/bitnami/zookeeper:3.8
ports:
- "2181:2181"
volumes:
- "zookeeper_data:/bitnami"
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
kafka:
container_name: kafka1
networks:
- "kafka_net"
image: docker.io/bitnami/kafka:3.4
ports:
#- "9092:9092" # this port not exposed !!!
- 29092:29092
volumes:
- "kafka_data:/bitnami"
environment:
KAFKA_CFG_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
ALLOW_PLAINTEXT_LISTENER: yes
depends_on:
- zookeeper
volumes:
zookeeper_data:
driver: local
kafka_data:
driver: local
networks:
kafka_net:
name: kafka_net
Rust Dev Container
version: "3.8"
services:
dev:
image: "rust:1.69-bookworm"
volumes:
environment:
- USER=debian
networks:
kafka_net:
driver: bridge # not sure if required?
ipam:
driver: default # again, not sure if required?
Docker Host (Debian Linux)
I don't fully understand the port specifications shown below. In particular kafka
appears to be doing something with both port 9092
and 29092
. But I don't understand any detail of what is being done here.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
27da034414c2 bitnami/kafka:3.4 "/opt/bitnami/script…" 15 hours ago Up 15 hours 9092/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp kafka1
54e31cea5355 bitnami/zookeeper:3.8 "/opt/bitnami/script…" 15 hours ago Up 15 hours 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp zookeeper1
521d9826af67 rust:1.69-bookworm "bash" 15 hours ago Up 15 hours rustdev
Finally, here are some network details. I am again unsure of why there are so many networks listed here, or if they are all in use or some of them are no longer used.
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
31f197ca5e13 bridge bridge local
ee0da54b7405 host host local
d171c50e5b6c kafka_net bridge local
40f877800e62 none null local
f92e84e1546d rust_default bridge local
Although my primary question is "Why can't I connect to Kafka via port 29092, and why can I connect via port 9092?" there are obviously several things which I am uncertain about here and so any direction regarding any of the above points raised is appreciated.