0

I am very new to docker and Kafka, and have a simple kafka python publisher shown below

The following are in my dockerfile:

FROM python:3.10

WORKDIR /app

COPY . /app

RUN pip install --user pip==23.0.1 && pip install pipenv && pipenv install --system

ENV ENVIRONMENT=production
CMD ["python3", "src/consumer.py"]

as well as my yaml file for compose:

version: '3'

services:
  zookeeper:
    image: confluent/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"

  kafka:
    image: confluent/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  publisher:
    container_name: publisher
    build:
      context: ./publisher
      dockerfile: Dockerfile
    environment:
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      KAFKA_TOPIC: metrics
      KAFKA_BROKER: kafka:9092
    depends_on:
      - kafka

  consumer:
    container_name: consumer
    build:
      context: ./consumer
      dockerfile: Dockerfile
    environment:
      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
      KAFKA_TOPIC: test_topic
      KAFKA_BROKER: kafka:9092
    depends_on:
      - kafka

in consumer.py I have:.

print('-- consumer script -- ')
import json
from kafka import KafkaProducer, KafkaConsumer

test_topic = "test_topic"
consumer = KafkaConsumer(auto_offset_reset='earliest', bootstrap_servers=['kafka:9092'],
                         api_version=(0, 10), consumer_timeout_ms=10000)
consumer.subscribe([test_topic])

for message in consumer:  # the line that seems to influence docker attachement
    print('-----loop works: ', message)

In one terminal I frist run:

docker-compose -f docker-compose.yml up zookeeper

and similarly for Kafka:

docker-compose -f docker-compose.yml up kafka

and then:

docker-compose -f docker-compose.yml up consumer

But the terminal seems stuck with the following lines:

Starting consumer ... done
Attaching to consumer

When I remove the line for message in consumer, the code follows without issue

Alejandro
  • 879
  • 11
  • 27
  • Does [Python app does not print anything when running detached in docker](https://stackoverflow.com/questions/29663459/python-app-does-not-print-anything-when-running-detached-in-docker) describe your situation? – David Maze Apr 09 '23 at 15:53
  • @DavidMaze I don't think so because when I remove the for loop line as noted in the question, everything else gets printed, but only when I have it, it seems stuck at attaching to .... – Alejandro Apr 09 '23 at 16:03
  • Unrelated to the problem, but `confluent/*` images are not correct, nor maintained. You want `confluentinc/*` – OneCricketeer Apr 10 '23 at 06:23

1 Answers1

0

In Compose, depends_on: kafka does not wait for the container to start.

Therefore, your consumer is possibly waiting to time out, while the kafka container takes a few seconds at least to start up. Therefore, add sleep(10), for example to the consumer

Secondly, no topics are automatically created by the Kafka container, therefore, your consumer is polling with no data to consume and "looks stuck", when it is actually just idling, which you could find if you privded group_id config to the consumer and ran kafka-consumer-groups --describe for that group

Sidenote: Kafka will default to only listen locally (at least bare metal install), so I always suggest you update the bind address

KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245