0

I have written two test applications which interface to Kafka. One of those applications is containerized inside a Docker container. Kafka is also containerized, running inside a Docker container. The specific Kafka image used is the Bitnami one.

The other application runs on the same machine (a physical Linux machine) which hosts the Docker containers.

Here is a description of the two applications:

  1. A Java application which runs on the docker host. (Same machine as the machine for which docker ps, etc, displays information about currently running containers.) This application was able to network with the containerized Kafka container via port 29092. This is the only port which is declared in the docker-compose.yml file for the Kafka container.

2: A Rust applicaiton which runs in its own Dockerized development environment. It is "adjacent" to the Dockerized Kafka instance, and both containers are connected to the same Docker network. (Or at least they should be.) This application does not work and cannot connect to Kafka via port 29092. I do not understand why. However, it is able to connect to Kafka via port 9092. Again, I do not understand why this is the case.


Here are some further details of the configuration:

Kafka docker-compose.yml

I don't fully understand what EXTERNAL_SAME_HOST and INTERNAL is doing in this configuration. What I do understand is that these are some kind of environment variable which the Kafka process/application picks up when it first starts and causes a socket to start listening to the specified address range and ports. Quite what the difference between "EXTERNAL_SAME_HOST" and "INTERNAL" is I am not sure. These strings seem to vary quite a bit depending on different examples, so I am not sure if they are arbitrary or what the possible values are.

version: "2"

services:
  zookeeper:
    container_name: zookeeper1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    container_name: kafka1
    networks:
      - "kafka_net"
    image: docker.io/bitnami/kafka:3.4
    ports:
      #- "9092:9092" # this port not exposed !!!
      - 29092:29092
    volumes:
      - "kafka_data:/bitnami"
    environment:
      KAFKA_CFG_LISTENERS: EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka1:9092,EXTERNAL_SAME_HOST://localhost:29092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper:2181"
      ALLOW_PLAINTEXT_LISTENER: yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

networks:
  kafka_net:
    name: kafka_net

Rust Dev Container

version: "3.8"

services:
  dev:
    image: "rust:1.69-bookworm"
    volumes:
    environment:
      - USER=debian

networks:
  kafka_net:
    driver: bridge # not sure if required?
    ipam:
      driver: default # again, not sure if required?

Docker Host (Debian Linux)

I don't fully understand the port specifications shown below. In particular kafka appears to be doing something with both port 9092 and 29092. But I don't understand any detail of what is being done here.

$ docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED        STATUS        PORTS                                                                     NAMES
27da034414c2   bitnami/kafka:3.4       "/opt/bitnami/script…"   15 hours ago   Up 15 hours   9092/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp                   kafka1
54e31cea5355   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   15 hours ago   Up 15 hours   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   zookeeper1
521d9826af67   rust:1.69-bookworm      "bash"                   15 hours ago   Up 15 hours                                                                             rustdev

Finally, here are some network details. I am again unsure of why there are so many networks listed here, or if they are all in use or some of them are no longer used.

$ docker network ls
NETWORK ID     NAME           DRIVER    SCOPE
31f197ca5e13   bridge         bridge    local
ee0da54b7405   host           host      local
d171c50e5b6c   kafka_net      bridge    local
40f877800e62   none           null      local
f92e84e1546d   rust_default   bridge    local

Although my primary question is "Why can't I connect to Kafka via port 29092, and why can I connect via port 9092?" there are obviously several things which I am uncertain about here and so any direction regarding any of the above points raised is appreciated.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
  • According to an answer here, the names "INTERNAL", "EXTERNAL" etc, are arbitrary https://stackoverflow.com/questions/53945143/internal-and-external-communication-in-kafka – FreelanceConsultant Apr 22 '23 at 11:31
  • That's correct. They have no explicit meaning as long as they are defined in the protocol mapping to real protocols defined by broker source code (PLAINTEXT, SSL, etc) – OneCricketeer Apr 22 '23 at 12:29
  • @OneCricketeer Why have you closed this as a duplicate? The linked question asks how to do something, here I have asked why one port works and not the other. Quite obviously I am not going to be able to get the information I have asked for from that other question. – FreelanceConsultant Apr 22 '23 at 13:00
  • 9092 works because you've advertised it within the bridge. The hostname is what matters, actually... Localhost doesn't work because it advertises that address to other clients, making them try to connect with themselves... My answer in the other question explains how the connection works between containers. You can also read the Confluent blog linked in the answer for more information – OneCricketeer Apr 22 '23 at 13:15
  • It's also written here for the container you're using https://github.com/bitnami/containers/tree/main/bitnami/kafka#accessing-apache-kafka-with-internal-and-external-clients – OneCricketeer Apr 22 '23 at 13:17
  • @OneCricketeer How does one determine what networks/hosts can connect to the Kafka application? This is what I'm struggling to get my head around. – FreelanceConsultant Apr 23 '23 at 17:04
  • `listeners` sets the BIND address. Replace mysql with Kafka in [answer here](https://stackoverflow.com/questions/3552680/bind-address-and-mysql-server) and it's the same logic – OneCricketeer Apr 24 '23 at 14:25

0 Answers0