How to use Kafka with docker and python?

Question

I am trying to set up a connection to Kafka using Docker. When running the code locally everything is working correctly, but when trying to run it using Docker Compose, I am getting the following error:

Traceback (most recent call last):
File "alerts_monitor_service.py", line 23, in <module>
     collect_data_from_db_and_alert()
File "alerts_monitor_service.py", line 11, in collect_data_from_db_and_alert
     kafka_dic = kafka_set_up()
File "/monitor_service/src/logic.py", line 49, in kafka_set_up
    producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
File "/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py", line 381, in __init__
    client = KafkaClient(metrics=self._metrics, metric_group_prefix='producer',
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 244, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 900, in check_version
    raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable

This is how I setup kafka in my docker-compose.yaml file:

version: "3.6"
services:
    zookeeper:
        image: wurstmeister/zookeeper
        ports:
          - 2181:2181
        depends_on:
          - mongo

    kafka:
        image: wurstmeister/kafka
        ports:
          - 9092:9092
        environment:
          KAFKA_ADVERTISED_HOST_NAME: localhost
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
        depends_on:
          - mongo
          - zookeeper

    mongo:
        container_name: mongo_db
        image: mongo:latest
        ports:
          - "27017:27017"
        volumes:
          - "./data/mongo:/data/db"

This is where I am getting the bug:

producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
producer = KafkaProducer()
producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))

1) `KAFKA_ADVERTISED_HOST_NAME` needs to be the Docker service name 2) Unclear where the Python code is running, but if its in a container, you should put it in the same compose file — OneCricketeer, Jun 21 '21 at 20:57

score 0 · Answer 1 · answered Jun 21 '21 at 14:27

Here is example Dockerfile: https://github.com/up9inc/async-ms-demo/tree/main/kafka

You can build it as docker build . -t kafka and then run locally as docker run -it -p 9092:9092 kafka. And then it would work for you locally.

For docker-compose, you would need the same Docker image ran with KAFKA_ADVERTISED_LISTENERS env var set, to let other hosts find it correctly. See the example here: https://github.com/up9inc/async-ms-demo/blob/main/docker-compose.yml#L39

How to use Kafka with docker and python?

1 Answers1