0

I started a kafka service using docker compose:

version: "2"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports:
      - 2181:2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes

  kafka:
    image: 'bitnami/kafka:2.4.1'
    container_name: kafka
    ports:
      - 29092:9092
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
      - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=100
    depends_on:
      - zookeeper

Kafka can be started properly and I could access kafka from other service containers defined in the same docker-compose.yml

However when I tried to connect the kafka from an offsetexplorer outside of the docker compose. I've got an error that the broker defined as localhost:29092 is not reachable. zookeeper host is ok.

What I'm trying to accomplish is:

Here's the dilemma, port 9092 is used for other services to access kafka (broker specified as kafka:9092). I need another advertised listener to allow access from e.g. local host instead of services. In this case, I need a different port, I was trying to map the host port 29092 to container port 9092 but it seems like it does not work.


I've found a solution, On the KAFKA_CFG_LISTENERS line: 0.0.0.0 is the key to make it work, localhost or 127.0.0.1 does not:

Kafka access inside and outside docker Follow the above question, I changed my docker-compose.yml file as following and now I can access kafka in side container from offset explorer running on local host outside of the docker compose network.

 kafka:
    image: 'bitnami/kafka:2.4.1'
    container_name: kafka
    ports:
      - '29092:29092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092
      - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,OUTSIDE://localhost:29092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper
Jin Ma
  • 169
  • 2
  • 12
  • Here's the dilemma, port 9092 is used for other services to access kafka (broker specified as kafka:9092). I need another advertised listener to allow access from e.g. local host instead of services. In this case, I need a different port, I was trying to map the host port 29092 to container port 9092 but it seems like it does not work. – Jin Ma Sep 14 '22 at 20:56
  • Actually, please ignore my last comment. I was reading too quickly and read the configuration backwards. Sorry about that! – larsks Sep 14 '22 at 21:06
  • Your ports are wrong. Needs `29092:29092` – OneCricketeer Sep 15 '22 at 18:02
  • no it does not work – Jin Ma Sep 15 '22 at 18:03
  • I posted the answer inside the question. – Jin Ma Sep 15 '22 at 18:07

0 Answers0