How to solve kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs?

Question

My docker-compose.yml code is:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

and my dockerfile is

FROM python
MAINTAINER Shubham Joshi 
ADD hello.py /
ADD transactions_producer.py /
COPY requirements.txt .
RUN pip3 install -r requirements.txt
CMD ["python","./transactions_producer.py"]

python code for the producer.py is:

from kafka import KafkaProducer
from time import sleep
from kafka.errors import KafkaError
import json

producer = KafkaProducer(bootstrap_servers=['localhost:9092'],api_version=(0, 10, 0),value_serializer=lambda v: json.dumps(v).encode('utf-8'))


for i in range(10):
    producer.send('shubham', create_random_transaction())
    sleep(5)
    print("Success",i)

producer.close()

shubham is the topic I created but I am not able to produce/publish messages in the topic and getting this timeout error.

step 1: Created an image:

docker build -t python_producer2 .

it built successfully, the I ran

step 2:docker run python_producer2

this is where I am getting this error

Traceback (most recent call last):
  File "./transactions_producer.py", line 52, in <module>
    producer.send('shubham', create_random_transaction())
  File "/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py", line 564, in send
    self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0)
  File "/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py", line 690, in _wait_on_metadata
    raise Errors.KafkaTimeoutError(
kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.

and how are you running your python Docker client? As part of Docker Compose? — Robin Moffatt, Jun 05 '20 at 10:57
added above in question, the code I ran on local it working fine and able to produce messages to kafka on my local machine, but getting timeout in docker — Shubham Joshi, Jun 05 '20 at 11:08
and how are you running your python Docker client? As part of Docker Compose? — Robin Moffatt, Jun 05 '20 at 11:36
in my requirements.txt I have added kafka-python, so it's installed when I am building my image for this build, Installing collected packages: kafka-python Successfully installed kafka-python-2.0.1 — Shubham Joshi, Jun 05 '20 at 11:52
That's not quite what I meant. You've listed a Dockerfile in your question. Are you using `docker run` or Docker Compose to execute it in the scenario in which it doesn't work? Can you update your question with details of this please. — Robin Moffatt, Jun 05 '20 at 11:55

score 4 · Accepted Answer · answered Jun 05 '20 at 13:14

When your code runs in Docker, localhost means the Docker container itself. This is why your connection times out, since Kafka isn't running on the container.

The first thing to do is get your container on the same Docker network as the broker. Either add it in to your Docker Compose, or specify the Docker Compose's network when you do docker run. The former is probably easier:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  client:
    image: python_producer2
    container_name: python_producer2
    depends_on: 
      - kafka

However, since your client is connecting to Kafka on a different server (since the containers are in effect separate servers), it can't use localhost, and nor can it connect to the listener on the broker that will return localhost as the advertised.listener. Therefore, change your code to:

producer = KafkaProducer(bootstrap_servers=['kafka:29092'],api_version=(0, 10, 0),value_serializer=lambda v: json.dumps(v).encode('utf-8'))

This will tell it to find the broker at kafka:29092, and the listener on 29092 which exposes kafka:29092 as the advertised.listener. To learn more about advertised listeners see https://rmoff.net/2018/08/02/kafka-listeners-explained/

its working now via docker-compose, thank you so much! – Shubham Joshi Jun 05 '20 at 13:51 — Shubham Joshi, Jun 05 '20 at 13:51

How to solve kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs?

1 Answers1