Publish Kafka objects in batches

Question

I am using KafkaProducer.send() method to publish records to Kafka. This is an asynchronous method. My application is publishing about 20k records to Kafka. A lot of times after successfully sending about 10k records I get Timeout Exception. I increased batch.size and linger.ms which reduced the problem but i still get Timeout Exception sometimes. Is there a way to send only 10k records at a time wait for them to complete and then send the next batch?? I do not want to make it synchronous by using send.get() because that will make it very slow. This is the exception I am getting.

Error publishing object on partition 0 org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s)

Did you try to use flush()? This will block your producer and make all the messages "send ready". — Manoj Vadehra, May 21 '19 at 09:09
Here's another question with exactly same exception: https://stackoverflow.com/questions/46649748/kafka-producer-timeoutexception-expiring-1-records — Manoj Vadehra, May 21 '19 at 09:10
I am using flush() after sending every message. I do not think that is working for me. — Anuja Barve, May 21 '19 at 09:13

score 0 · Answer 1 · answered May 21 '19 at 13:31

0

Did you try set retries=3 to the producer config?
And probably in case of record expiration you should not increase batch.size but decrease it.

answered May 21 '19 at 13:31

mishak471

195
5

Yes, *smaller* batches seem better if you run into timeouts. And be aware that retries can result in messages being produced out of order, which may or may not be a problem for your application. – Thilo May 21 '19 at 13:34
@Thilo How do I send it in batches?? – Anuja Barve May 28 '19 at 12:34
The producer does this automatically, according to configuration (such as `batch.size`) – Thilo May 28 '19 at 12:54

Publish Kafka objects in batches

1 Answers1