Kafka Timeout Exception:Batch Expired

Question

We are facing following issue on production:

org.apache.kafka.common.errors.TimeoutException: Batch Expired.

Is it because of invalid configuration like batch size, request timeout or any other thing?

Possible duplicate: https://stackoverflow.com/questions/34794260/when-does-the-apache-kafka-client-throw-a-batch-expired-exception — OK sure, Oct 01 '18 at 12:24
Please read [Under what circumstances may I add “urgent” or other similar phrases to my question, in order to obtain faster answers?](//meta.stackoverflow.com/q/326569) - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions. — halfer, Oct 01 '18 at 13:50

score 3 · Answer 1 · answered Oct 01 '18 at 12:28

3

The error indicates that some records are put into the queue at a faster rate than they can be sent from the client.

When your Producer sends messages, they are stored in buffer (before sending the to the target broker) and the records are grouped together into batches in order to increase throughput. When a new record is added to the batch, it must be sent within a -configurable- time window which is controlled by request.timeout.ms (the default is set to 30 seconds). If the batch is in the queue for longer time, a TimeoutException is thrown and the batch records will then be removed from the queue and won't be delivered to the broker.

Increasing the value of request.timeout.ms should do the trick for you.

answered Oct 01 '18 at 12:28

Giorgos Myrianthous

36,235
20
134
156

we already tried it out but it didn't work for us. What batch size should be preferred ? – user3392430 Oct 03 '18 at 08:50
@user3392430 It depends on the volume of your data. Note that if the `batch.size` is to small, the producer will need to send messages more frequently. By the way, did you restart your services after changing the configurations? – Giorgos Myrianthous Oct 03 '18 at 09:00
@user3392430 Did you also get a `ChannelClosedException`? – Giorgos Myrianthous Oct 03 '18 at 09:25
thanks for the response. Yes we restarted the services after making the configuration changes and didn't get ChannelClosedException – user3392430 Oct 03 '18 at 10:37
what are the other possible reasons which may cause this exception? – user3392430 Oct 03 '18 at 10:38
What is the value of linger.ms? – Giorgos Myrianthous Oct 03 '18 at 10:39

Kafka Timeout Exception:Batch Expired

1 Answers1