Kafka access inside and outside docker

Question

I'm trying to start a kafka service using docker-compose, and it should be able to be accessed inside and outside docker. So, it should be matter of setting the right advertisers inside and outside:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9094:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://127.0.0.1:9094
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://127.0.0.1:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

The problem is that when I try to connect from outside the cluster, I don't get 127.0.0.1 as the name of the node, but the internal hostname:

$ kafkacat -L -b 127.0.0.1:9094
Metadata for all topics (from broker -1: 127.0.0.1:9092/bootstrap):
 1 brokers:
  broker 1001 at 91588ea968d4:9092
 28 topics:
 ...

Isn't the purpose of KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS to handle that situation? I tried setting KAFKA_ADVERTISED_HOST_NAME but it's ignored (one piece of documentation says it's deprecated, other one says that it's still active), but nevertheless that doesn't seem to be the answer, since I want two different advertised hostnames for two different networks.

I guess the old question remains: how to make kafka work inside and outside docker-compose?

Can you link to the two docs you mention that are inconsistent? It would be good to fix them if there is an ambiguity. — Robin Moffatt, Nov 11 '18 at 11:47
It's in https://hub.docker.com/r/wurstmeister/kafka/ section "listener configuration". It's not kafka documentation, but wurstmeister docker image: " Later versions of Kafka have deprecated advertised.host.name and advertised.port. NOTE: advertised.host.name and advertised.port still work as expected, but should not be used if configuring the listeners." — jdinunzio, Nov 11 '18 at 12:37
Ah right. FWIW Confluent have a full set of Kafka images here: https://hub.docker.com/u/confluentinc/ — Robin Moffatt, Nov 11 '18 at 12:51

score 43 · Answer 1 · edited Jan 13 '22 at 15:33

43

What I needed to do was to declare the LISTENERS as both binding to all interfaces, and then advertise them differently - one to the docker network, one to the host:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
  kafka:
    image: confluentinc/cp-kafka
    ports:
      - 9094:9094
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

edited Jan 13 '22 at 15:33

Tim

19,793
8
70
95

answered May 22 '19 at 10:33

daniu

14,137
4
32
53

3

There is a colon instead of a dot in KAFKA_LISTENERS : INTERNAL://0.0 **:** 0.0:9092,OUTSIDE://0.0.0.0:9094 should be: INTERNAL://0.0.0.0.9092,OUTSIDE://0.0.0.0:9094 – jean-loup Monnier Dec 11 '21 at 17:57
When I try KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092, my Kafka fails to start with "Unknown Hostexception": kafka. How do I fix this? – Anand S Aug 27 '22 at 05:57
how can i connect to msk from this ? i added aws msk arn to the KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS , but i cant conenct it keeps getting timeout – Sithija Piyuman Thewa Hettige Oct 24 '22 at 07:30
``` KAFKA_ADVERTISED_LISTENERS:INTERNAL://kafka:9092,OUTSIDE://localhost:9094 ``` Mentioned here is the host name rather than service name – Hemanth Kumar Nov 18 '22 at 14:18
worked great to connect to local bitnami/kafka docker image with conduktor client. – JCompetence Jan 24 '23 at 09:06
I cannot connect, using the docker-compose.yml, getting error: kcat -L -b 127.0.0.1:9094 %3|1687154329.580|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9094/bootstrap]: 127.0.0.1:9094/bootstrap: Connect to ipv4#127.0.0.1:9094 failed: Connection refused (after 0ms in state CONNECT) %3|1687154330.584|FAIL|rdkafka#producer-1| [thrd:127.0.0.1:9094/bootstrap]: 127.0.0.1:9094/bootstrap: Connect to ipv4#127.0.0.1:9094 failed: Connection refused % ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m ?) – kamal Jun 19 '23 at 06:00

score 22 · Accepted Answer · answered Nov 11 '18 at 11:46

22

Your config of the listeners looks correct, the problem is with your Docker Compose:

ports:
  - "9094:9092"

You’re mapping 9094 (‘Outside’) back to 9092 (‘Inside’) and thus when you connect you’re connecting to the ‘Inside’ listener. If you remove this line of config then your listener set up should work as intended.

For more info, see http://rmoff.net/2018/08/02/kafka-listeners-explained/

answered Nov 11 '18 at 11:46

Robin Moffatt

30,382
3
65
92

I have read that post so many times by now, but I still can't get it to work. If I use the docker-compose snippet under the "HOW TO", it gives the error "Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS". So I add the corresponding KAFKA_LISTENERS, and it says "Each listener must have a different port". I change the external port to 9094, and I can't connect to localhost:9094, but to :9092, with the "broker address" metadata "kafka9:9092", so sending a message fails. – daniu May 22 '19 at 09:26
@daniu start a new question here on StackOverflow with your Docker Compose file and the error you get when you use it, and we can fix it from there. – Robin Moffatt May 22 '19 at 09:55
4

Had nearly finished writing it when I figured it out. Your post is missing the most relevant line, namely the KAFKA_LISTENERS. You need to set both to the kafka host here ("INTERNAL://kafka:9092,OUTSIDE://kafka:9094"), and then can advertise them differently ("INTERNAL://kafka:9092,OUTSIDE://localhost:9094"). – daniu May 22 '19 at 10:30
@daniu I'd be happy to add it - which section is it missing from? the "HOWTO: Connecting to Kafka on Docker" ? – Robin Moffatt May 22 '19 at 10:48
yep, the docker-compose snippet, below `environment` (the values of which are not indented correctly either btw). – daniu May 22 '19 at 10:52
@daniu I've updated it—LMK if it is clearer now. you can email/tweet me if it's easier. – Robin Moffatt May 22 '19 at 12:56

score 4 · Answer 3 · answered Apr 07 '19 at 16:17

Agreed to the answer from @Robin Moffatt that outside port 9094 is mapped to container's 9092.

In addition to removing that configuration from docker-compose.yml, the outside port 9094 should be mapped to container's 9094.

ports:
  - "9094:9094"

PS: I couldn't add this as a comment to previous answer, hence adding it as a new answer.

Kafka access inside and outside docker

3 Answers3

Linked

Related