I am trying to connect to Kafka using Python and I am able to connect and see the topic names however when I try to read messages for any topic nothing happens. It feels like the connection times-out as python does not throw an error.
I am port-forwarding to Kafka and then using localhost:{port} as my bootstrap_server.
I do not have experience working with Kubernetes, Kafka or anything networking or cloud related and the person who set-up our system up has left the company.
I am using the kafka-python package (version 2.0.2) and python 3.9. The simple code below (basically copied from the docs) is being run within Pycharm, no docker container.
from kafka import KafkaConsumer
consumer = KafkaConsumer('my-topic', bootstrap_servers='localhost:10523')
# this works
topics = list(consumer.topics())
# this does not work
for msg in consumer:
print(msg)
The Kafka topic already contains ~10,000 messages that I thought should by printed by the code above. I have also tried using auto_offset_reset='earliest' but that also does not work.
When the Kafka messages do not print my port-forward does throw this error (after I close my python console), E0420 20:36:09.333793 26252 portforward.go:385] error copying from local connection to remote stream: read tcp6 [::1]:10523->[::1]:123456: wsarecv: An existing connection was forcibly closed by the remote host.
Is there something wrong with the Kafka setup? How could I check this?
How could I debug to locate the actual issue?
Should I be port-forwarding to connect or is there a better way to connect using my local machine?
