0

I am aware this topic has several questions and blog posts about it. I am following these two: https://rmoff.net/2018/08/02/kafka-listeners-explained/

https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/ But unfortunately, without success. I'm trying to make it so the same code will work whether I'm running it from my IDE where the kafka client is in a container, or whether the code I'm running is in a container within the network. I am able to make each scenario work on its own, but not the two together. My docker compose:

  zoo1:
    image: confluentinc/cp-zookeeper:7.2.1
    hostname: zoo1
    container_name: zoo1
    ports:
     - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:7.2.1
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
    depends_on:
      - zoo1

In this docker-compose, communication within the docker network using kafka1:29092 as bootstrap works great. but, from my laptop using the same doesn't work. Is there anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092? Do I even need the external listener? Thanks

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Omri. B
  • 375
  • 1
  • 13
  • We need to distinguish between internal and external connections since the actual kafka hosts are accessible through different names (`kafka1:29092` internally, `localhost:9092` externally). Thus, the address to the bootstrap server must be different, depending on the runtime environment (internally vs. externally). And yes, we need the internal and external mapping for this to work. – Turing85 Sep 04 '22 at 07:22
  • @Turing85 thanks, but shouldn't the port mapping make it possible for me to communicate using `kafka1:29092` even from my IDE? is there any way to achieve my desired functionality (that isn't a hack)? – Omri. B Sep 04 '22 at 07:25
  • The initial connection isn't the problem; the advertised listener to connect to the kafka-cluster is (internally, it must resolve to `kafka1`, externally it must resolve to `localhost`). And yes there is. You have it right there. [I do something very similar in my local setup(`bitbucket.org`)](https://bitbucket.org/turing85/uber-docker-compose/src/73909d6f52a9f6edb357183b867e64bf2931812d/localdeployment/docker-compose.yml#lines-83). – Turing85 Sep 04 '22 at 07:28
  • Thanks for your reference, I'll take a look. My intention was - no way to bootstrap to the same address both internally and externally? I'll have to change to `localhost:9092` on my IDE? – Omri. B Sep 04 '22 at 07:31
  • 2
    I would suggest to [externalize (`12factors.net`)](https://12factor.net/config) the connection to the kafka bootstrap server, defaulting to whatever you need it to be in your IDE, and then override it through an environment variable in the docker deployment. – Turing85 Sep 04 '22 at 07:33
  • Maybe this can help you https://stackoverflow.com/questions/35828487/docker-1-10-access-a-container-by-its-hostname-from-a-host-machine If you are using Linux, you can also use --network=host, then access your kafka container with `localhost:29092` from inside and outside of the docker – Ron Serruya Sep 04 '22 at 07:40

1 Answers1

1

anyway to ensure that both locally and inside the container network I can bootstrap to kafka1:29092?

No.

Your host isn't aware of the DNS / service names used by Docker.

Instead, add an environment variable in your code like KAFKA_BOOTSTRAP_SERVERS, then set that as a variable in your IDE (as localhost:9092) via a run config, or as a container variable (kafka1:29092)

You can also remove - "29092:29092" from your compose file since your host will never need that port to connect with the broker

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245