0

I'm trying to use Debezium with Kafka connect, I followed this tutorial, and everything connected just fine. However, the problem is that I cannot access Kafka from outside of docker containers anymore.

I use these commands to start containers:

docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:2.0.0.Beta1

docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:2.0.0.Beta1

docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets --link kafka:kafka debezium/connect:2.0.0.Beta1

I tried to set KAFKA_ADVERTISED_LISTENERS to PLAINTEXT://127.0.0.1:9092 which allowed me to connect to Kafka from the outside of the container but I could not connect from connect container to kafka container anymore. How can I achieve both?

omidh
  • 2,526
  • 2
  • 20
  • 37
  • have you tried `docker inspect` on `connect` to find its external ip and setting that in `KAFKA_ADVERTISED_LISTENERS`. – whitespace Aug 14 '22 at 13:00

2 Answers2

1

with this you can access the kafka container from your host on the port 9092

  zookeeper:
    image: confluentinc/cp-zookeeper:7.2.0
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka-broker:
    image: confluentinc/cp-kafka:7.2.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
raphaelauv
  • 670
  • 1
  • 11
  • 22
-1

I think it's not a Kafka issue, but a docker network one. It's probably accessible via docker network or you need to expose it. https://docs.docker.com/network/network-tutorial-standalone/

themoah
  • 167
  • 2
  • 13
  • 2
    Yes, it is because, unlike other images, if you don't explicitly define `KAFKA_ADVERTISED_LISTENERS` you can't access it from outside of the container even if you set the `--port` option. But for example postgres image works just fine by `--port 5432:5432` config. – omidh Aug 14 '22 at 10:39
  • @omidh That setting is only necessary because of Zookeeper... In Kafka Kraft mode, a simple port forward should work – OneCricketeer Aug 15 '22 at 14:27