Setting up docker-compose ports and host names to work with a kafka-python container

Question

I'm learning Kafka at the moment, and struggling to get my docker-compose configuration set up properly. What I' trying to do is run a broker based on the wurstmeister/kafka image, and then another container that runs a simple python script with kafka-python.

I've been following this tutorial mostly, but I suspect my handling of the ports is a bit of a mess. Here's my docker-compose.yml:

version: '3'

services:

  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    expose:
      - "9093"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_CREATE_TOPICS: "client-pusher:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper

  app-python:
    build: .
    ports:
      - "9093:9093"
    expose:
      - "9093"
      - "9092"
    depends_on:
      - "kafka"

To tell the honest truth, I don't have a clue what I'm doing half the time when it come to ports in Docker.

Using this Dockerfile

FROM python:3.8

ENV PYTHONUNBUFFERED=1

# set the working directory in the container
WORKDIR /code

# copy the dependencies file to the working directory
COPY requirements.txt .

# install dependencies
RUN pip install -r requirements.txt

# copy the content of the local src directory to the working directory
COPY . .

# command to run on container start
CMD ["python","/code/consumer.py"]

I can make this script spit out some logs:

# consumer.py
import json
from datetime import date
from typing import Optional
import time
import logging


from kafka import KafkaConsumer
from pydantic import BaseModel


class Client(BaseModel):
    first_name: str
    email: str
    group_id: Optional[int] = None
    date: date


# consumer = KafkaConsumer(
#     'client-pusher',
#     bootstrap_servers=['kafka:9093'],
#     auto_offset_reset='earliest',
#     enable_auto_commit=True,
#     group_id='my-group-id',
#     value_deserializer=lambda x: json.loads(x.decode('utf-8'))
# )

count = 0
while True:
    # msg_pack = consumer.poll(timeout_ms=500)

    logging.warning(f"Hi there {count}")

    time.sleep(2)
    count += 1

    # for tp, messages in msg_pack.items():
    #     for message in messages:
    #         client = Client(**message.value)
    #         print(client)

but when the commented code is uncommented, the connection fails. The

bootstrap_servers=['kafka:9093'],

line results in

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

I feel like there's some magic combination of exposing or configuring the ports properly in the docker-compose file and using them properly in the python script, and/or configuring the service names properly. But I'm lost. Can anyone help?

Answer 1

TLDR; remove all expose's and adjust app-python's ports to something that isn't already referenced. In your code, instead of kafka:9093, use localhost:9092

Two things: i. For app-python, you're exposing your machines port 9093 (localhost:9093) to the containers port 9093 (app-python:9093). Both containers can't expose the same machine port, so i recommend keeping your kafka container(s) port config a comfortable distance from your apps port (maybe 9092/9093 for kafka + 8080 for your app)

ii. Docker compose puts all the containers listed in the file within the same network. So there's two way to go about it. If you want to run kafka in docker and your python code in your IDE/terminal, hardcode localhost:9092 in your python script. i.e. your code connects to kafka through its external port mapping (OUTSIDE)

If you run it like how you're running it now, with both containers running in the same docker network) I suggest passing an environment variable (or property you can pass in and reference in the code) to app-python with the bootstrap server <Container name>:<INSIDE PORT> --- kafka:9093

Here's an example that I have with Java, where I could run the app (rest) inside or outside docker-compose. If outside, i referenced localhost:9092 but if inside, I referenced like this:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

  kafka:
    container_name: kafka_broker_1
    image: wurstmeister/kafka
    links:
      - zookeeper
    ports:
      - "9092:9092"
      - "29092:29092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOSTNAME: kafka
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock%

  rest:
    image: rest:latest
    container_name: rest
    build:
      context: rest
      dockerfile: Dockerfile
    links:
      - kafka
    environment:
      - SPRING_KAFKA_BOOTSTRAP-SERVERS=kafka:29092
      - SERVER_PORT=8080
    ports:
      - "8080:8080"
    depends_on:
      - kafka

Answer 2

AFAIK expose is only informative (see here). It's all about the ports you define with ports.

Try to connect to the port you defined in ports (for inside and outside), i.e. in your case

bootstrap_servers=['kafka:9092']

And remove all occurrences of connecting to the ports defined as expose, e.g. for KAFKA_LISTENERS.