Kafka running in container - cannot connect from another container

Question

I've tried to follow the advice of some other excellent posts such as this one and others, but still running into problems.

I have two docker compose networks, one running kafka/zookeeper and another running a python application which includes a kafka producer, so needs to write to the container running kafka. I've created a network kafka_logging_network which both containers are a part of. I've double checked the KAFKA_ADVERTISED_LISTENERS variable many times but cant see what I'm missing.

The below shows the two docker-compose.yml files, followed by the Dockerfile and main.py for the application. main.py should run without error but in reality it cannot find any kafka brokers at the given KAFKA_HOST and KAFKA_PORT, raising kafka.errors.NoBrokersAvailable. Where have I gone wrong?

kafka/docker-compose.yml (for kafka/zookeeper)

version: '3.7'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888
    networks:
      - logging_network

  kafka1:
    image: confluentinc/cp-enterprise-kafka:5.3.1
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
    depends_on:
      - zoo1
    restart: always
    networks:
      - logging_network

networks:
  logging_network:
    driver: bridge

app_folder/docker-compose.yml (for python application)

version: '3.7'

services:
  pyapp:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: pyapp
    networks:
      - kafka_logging_network
    environment:
      KAFKA_HOST: kafka1
      KAFKA_PORT: 29092
      KAFKA_TOPIC: logs-topic

networks:
  kafka_logging_network:
    external: true

Dockerfile

FROM python:3.9.7
COPY . .
RUN pip install kafka_python==2.0.2
ENTRYPOINT python main.py

main.py

from kafka import KafkaProducer
import os

host = os.environ['KAFKA_HOST']
port = os.environ['KAFKA_PORT']

kp = KafkaProducer(bootstrap_servers=f'{host}:{port}')
print('success')

I think the port of kafka you are trying to connect in docker-compose is wrong. Can you please double check it? — Maninder, May 31 '23 at 14:42
It might be because you added the port `29092` in your advertised listeners, but you're trying to use `19092` in your application. — ndogac, May 31 '23 at 14:42
My bad - error in question. I actually have 29092 in advertised listeners and 29092 in application. — John F, May 31 '23 at 14:45
No @GuillaumeBarré - they are separate compose commands. So `docker-compose up` is run twice - once for Kafka, once for the python app. — John F, May 31 '23 at 14:49
Note: You should use Bootstrap server string as one app environment variable. In production, you shouldn't define just one host and port — OneCricketeer, May 31 '23 at 20:24

score 1 · Accepted Answer · answered May 31 '23 at 15:18

The way you have configured kafka and the client application is fine.

Actually you have just a problem with how your kafka is trying to access Zookeeper

environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'

You don't have a service named 'zookeeper' but 'zoo1'

Replace your configuration by :

environment:
  KAFKA_BROKER_ID: 1
  KAFKA_ZOOKEEPER_CONNECT: 'zoo1:2181'

Kafka running in container - cannot connect from another container

kafka/docker-compose.yml (for kafka/zookeeper)

app_folder/docker-compose.yml (for python application)

Dockerfile

main.py

1 Answers1