I have set up a AWS EC2 instance running Apache Kafka 0.8 via a Bitnami AMI image. The server properties are pretty much default (Kafka located at localhost:9092 and zookeeper located at localhost:2181).
When I SSH into the machine, I can produce/consume data using the scripts provided by Kafka, located at kafka/bin. To produce I run the following command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
To Consume:
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This works correctly, thus I have determined that Kafka is functioning correctly. Next I attempted to produce/consume from my machine, using the python library pykafka:
client = KafkaClient(hosts = KAFKA_HOST)
topic = client.topics[sys.argv[1]]
try:
with topic.get_producer(max_queued_messages=1, auto_start=True) as producer:
while True:
for i in range(10):
message = "Test message sent on: " + str(datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y"))
encoded_message = message.encode("utf-8")
mess = producer.produce(encoded_message)
except Exception as error:
print('Something went wrong; printing exception:')
print(error)
And I consume as follows:
client = KafkaClient(hosts = KAFKA_HOST)
topic = client.topics[sys.argv[1]]
try:
while True:
consumer = topic.get_simple_consumer(auto_start=True)
for message in consumer:
if message is not None:
print (message.offset, message.value)
except Exception as error:
print('Something went wrong; printing exception:')
print(error)
These snippets run without errors or exceptions, but no messages are produced or consumed, not even the ones created via the local scripts.
I have confirmed that both ports 9092 and 2181 are open via telnet. My questions are as follows:
- Is there a way to debug such problems and find the root cause? I would expect the library to throw an exception if there is some connectivity issues.
- What is going on?
