Why is producer in pykafka so slow?

Question

I wrote a simple producer using pykafka but can't seem to get it to perform. The basic producer and call to produce is below. When I call this 100 times with a small message, and add some timing/profiling code, it takes about 14 seconds. I understand this to be an asynchronous sending of messages so I would expect it to be incredibly fast. Is there some setting I'm missing? I've also tried it with min_queued_messages=1 and those takes about 2 seconds longer.

from pykafka import KafkaClient
import time

client = KafkaClient(hosts="kafka1.mydomain.com:9092", exclude_internal_topics=False)
topic = client.topics['mytopic']

start = time.time()

for x in xrange(100):
    with topic.get_producer(delivery_reports=False,
                            sync=True,
                            linger_ms=0) as producer:
        producer.produce("This is a message")

end = time.time()
print "Execution Time (ms): %s" % round((end - start) * 1000)

I did do a profile of this within pycharm and is says hat 78.8% of the time is spent on "time.sleep"?! Why would it be sleeping?

I am not sure this will help, but _[this link](https://github.com/Parsely/pykafka/issues/334)_ is discussing differences between `pykafka` and `kafka-python`, and their different handling of creating topics dynamically. — ryyker, Apr 02 '18 at 18:17
Looks like if I set "required_acks=0" then my run time drops to about 1500 ms! — GregH, Apr 02 '18 at 21:08

score 2 · Answer 1 · answered Apr 09 '18 at 23:23

The topic.get_producer call is meant to be called once at the beginning of the producer's lifespan. Calling it in a tight loop as your example code does will cause the initialization sequence to be run repeatedly, which is unnecessary and will add a lot of overhead. Your code would work faster if it were changed to the following:

with topic.get_producer(delivery_reports=False,
                        sync=True,
                        linger_ms=0) as producer:
    for x in xrange(100):
        producer.produce("This is a message")

Why is producer in pykafka so slow?

1 Answers1

Linked