0

I have a setup of Debezium which uses Kafka. I am able to consume messages from kafka console as described in the doc. However, when I create a kafka consumer using Python on my local, I am unable to consume messages. It should be noted that kafka console works just fine!

I tried looking into this issue but was unable to have similar environments/situations

My python code to connect is:

from kafka import KafkaConsumer
consumer = KafkaConsumer('dbserver1.inventory.customers', group_id='my-group', bootstrap_servers=['localhost:9092'], auto_offset_reset='earliest')
for message in consumer:
    print(message)

This just goes blank regardless of existing messages or new messages pushed to this topic.

I am sure that the messages exist because when I open up a console consumer, I get to see the messages.

Just to be clear on the whole setup: I have followed this (https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mongodb) doc for each step(except the last one). Everything works but my Python code. I also tried creating a consumer with kafka:9092 bootstrap server but it ends up in an error:

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

My local is Mac OS.

FYI: I am able to get everything else, like topics:

>>> consumer = KafkaConsumer('dbserver1.inventory.customers', group_id='my-group', bootstrap_servers=['localhost:9092'], auto_offset_reset='earliest')
>>> consumer.topics()
{'my_connect_offsets', 'my_connect_configs', 'dbserver1.inventory.orders', 'connect-status', 'dbserver1.inventory.customers', 'dbserver1.inventory.products'}

I am starting consumer via command:

docker-compose -f debezium-mongodb.yaml exec kafka /kafka/bin/kafka-console-consumer.sh \
    --bootstrap-server kafka:9092 \
    --from-beginning \
    --property print.key=true \
    --topic dbserver1.inventory.customers
Aviral Srivastava
  • 4,058
  • 8
  • 29
  • 81

1 Answers1

0

Without seeing your compose file, localhost:9092 likely will not work in your Python code based on your docker command

  1. If your Python code is not running in a container, it needs to read from a different port. If it is running in a container, you must use kafka:9092
  2. The port you use depends on the advertised listeners of the container

Connect to Kafka running in Docker from local machine

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How come new topics gets listed when the broker is `localhost:9092`? As for the docker file: https://github.com/debezium/debezium-examples/tree/master/tutorial#using-mongodb – Aviral Srivastava Apr 30 '19 at 03:41
  • 1
    The topic list comes from Zookeeper and does not use the Consumer API protocol. I have not used the `debezium/kafka` images, so I do not know how their networking is configured, but your Compose file isn't setting the advertised host to be anything external to the docker network https://github.com/debezium/docker-images/tree/master/kafka/0.10#advertised_host_name I will bet you that if you put your Python code in a container, then use `kafka:9092` as the connection string, it'll likely start working. – OneCricketeer Apr 30 '19 at 03:48
  • Otherwise, you're welcome to read this blog for more details than are worth explaining again here https://rmoff.net/2018/08/02/kafka-listeners-explained/ – OneCricketeer Apr 30 '19 at 03:49
  • Note: you do not need a `debezium/kafka` image at all... You can swap that out for any other Kafka image. – OneCricketeer Apr 30 '19 at 03:54
  • I read https://rmoff.net/2018/08/02/kafka-listeners-explained/ and configured my `yaml` file as shown here(https://pastebin.com/SeDZ4aiG) I am getting an error: https://pastebin.com/6w5CTwGs Could you see the yaml file and correct me if my configs are wrong? – Aviral Srivastava Apr 30 '19 at 04:34
  • You need two listeners. And it doesn't have to be "Fred" :) You only have one, that now says `localhost`, which mean that Debezium won't be able to reach `kafka:9092` anymore. You need one listen internal to the Docker network like `DOCKER://kafka:9092`, then another like `DOCKER_HOST://localhost:29092`, then you use different ports just because Kafka can't advertise two listeners on the same port. See my answer here https://stackoverflow.com/a/51634499/2308683 – OneCricketeer Apr 30 '19 at 04:44