Unable for .NET client to send messages to Kafka on Docker

Question

I have set up Zookeeper and Apache Kafka on Docker (Windows) using the following docker-compose.yml:

version: '2'
services:

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181
    extra_hosts:
      - "localhost: 127.0.0.1"
    networks:
      - app-network
  
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - 9094:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    extra_hosts:
      - "localhost: 127.0.0.1"
    networks:
      - app-network
      
networks:
  app-network:
    driver: bridge

Docker containers are up and running, and verified Kafka's external listener via telnet (telnet localhost 9094). I have followed Confluent's guide on setting these up - https://www.confluent.io/blog/kafka-listeners-explained/. I also have verified Kafka's external connectivity via clients such as https://www.kafkamagic.com/.

However I have my own .NET client that is trying to connect to Kafka via localhost:9094 using Confluent.Kafka NuGet package and is throwing the following errors:

%3|1630480353.603|FAIL|rdkafka#producer-3| [thrd:kafka:9092/1]: kafka:9092/1: Failed to resolve 'kafka:9092': No such host is known.  (after 2343ms in state CONNECT)
%3|1630480353.603|ERROR|rdkafka#producer-3| [thrd:app]: rdkafka#producer-3: kafka:9092/1: Failed to resolve 'kafka:9092': No such host is known.  (after 2343ms in state CONNECT)
%3|1630480356.907|FAIL|rdkafka#producer-3| [thrd:kafka:9092/1]: kafka:9092/1: Failed to resolve 'kafka:9092': No such host is known.  (after 2295ms in state CONNECT, 1 identical error(s) suppressed)
%3|1630480356.907|ERROR|rdkafka#producer-3| [thrd:app]: rdkafka#producer-3: kafka:9092/1: Failed to resolve 'kafka:9092': No such host is known.  (after 2295ms in state CONNECT, 1 identical error(s) suppressed)

How come the errors are showing connectivity errors on :9092 if I specified the broker as :9094? Is there something wrong with Kafka's setup on docker-compose file?

Answer 1

Managed to get it working by modifying the docker-compose.yml specifically kafka's ports and KAFKA_ADVERTISED_LISTENERS; and removed KAFKA_LISTENERS.

version: '2'
services:
  
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181
    extra_hosts:
      - "localhost: 127.0.0.1"
    networks:
      - app-network
  
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    extra_hosts:
      - "localhost: 127.0.0.1"
    networks:
      - app-network
      
networks:
  app-network:
    driver: bridge