0

I have Kafka running on a container on my desktop. I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example.

I'm having issues reading and writing to/from a Kafka topic. what's annoying me is that it won't give me an error message, it's behaving like if the topic doesn't have any messages on it, but it does, I can see using the tool, even added two messages manually.

the topic exists and has two messages on it (which I added manually using this UI)

problem: the code that sends messages to the topic runs fine, but the messages don't make it to Kafka the code that reads messages from the topic doesn't read anything. It sits there like if there are no messages to be read. Also, I can use the same consume to list the topics (which indicates the connection was successful)

The kafka version is 2.4. Any idea what the problem may be? I have tried "bootstrap_servers=['localhost:9092', 'kafka-server:9092']" but it also didnt work

Thanks

enter image description here

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Diego
  • 34,802
  • 21
  • 91
  • 134

1 Answers1

1
  1. KafkaProducer: You need to execute flush after send
producer.send('testTopic', b'Hello, World!')
producer.flush()
  1. KafkaConsumer: Specify bootstrap_servers and auto_offset_reset
consumer = KafkaConsumer('testTopic',
                         bootstrap_servers=['localhost:9092'],
                         auto_offset_reset='earliest')
for message in consumer:
    print(message)
ywbaek
  • 2,971
  • 3
  • 9
  • 28
  • hey @oldwooki I was specifying the bootstrap_servers on the consumer and I tried to flush on the producers, which didnt help. It turns out that I had to use port 29092, not 9092... any idea why? I tought 9092 was the default – Diego Mar 21 '20 at 19:24
  • @Diego Are you trying to access the kafka from other containers? – ywbaek Mar 21 '20 at 19:35
  • no, just from my local desktop. But Kafka is running on a container, which has both two ports mapped – Diego Mar 21 '20 at 19:48
  • @Diego then that's a docker config problem, not python. 9092 is the default... – OneCricketeer Mar 22 '20 at 00:14