0

I am trying to send 4000 messages(Json format to KAFKA. But after sending around 3900 messages every time I get the below error. How can I avoid getting the below error?

I am using JsonSerialiser for Kafka value .

org.apache.kafka.common.errors.TimeoutException: Expiring 11 record(s) for <topic-name_here> : 121046 ms has passed since batch creation

Below is my Kafka configuration settings

acks==allretries=3
retry.backoff.ms=1500
request.timeout.ms= 200000  // 200 sec
linger.ms= 3000 ms  // 30 sec
batch.size=32000
compression.type=gzip

Below is my code snippet for Kafka OnSucess() method for Kafka send. I am using an asynchronous call I tried tweaking batch.size ,buferr.memory and max.block.ms but still getting the same error.

@Overridepublic 
void onSuccess(SendResult<String, Person> result)
 {      
//SendResult to create a java object for each message   
//Call a stored proc to save the object in database for each message

}

Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
  • Does this answer your question? [Kafka producer TimeoutException: Expiring 1 record(s)](https://stackoverflow.com/questions/46649748/kafka-producer-timeoutexception-expiring-1-records) – Shrey Jakhmola Sep 17 '20 at 19:24
  • I don't know if this is the problem but the docs for `delivery.timeout.ms` says `>The value of this config should be greater than or equal to the sum of request.timeout.ms and linger.ms.` It's default is 120,000 and you have increased `request.timeout.ms+linger.ms` beyond that. The `deliveryTimeout` is used to determine expired batches. – Gary Russell Sep 17 '20 at 19:25
  • let me reduce the value of request.timeout.ms – Pale Blue Dot Sep 17 '20 at 19:26
  • @GaryRussell No it doesnt solve the problem.I gave` linger.ms=5 and request.timeout.ms=60,000` and commented `batch.size` but still got the same error – Pale Blue Dot Sep 17 '20 at 19:44
  • @GaryRussell i was getting the above error due to database call in callback.I still don't know the cause of the error but i solved it by executing call to DB (in callback) by executing it another thread as you had mentioned in one of your previous answer.Thanks – Pale Blue Dot Sep 18 '20 at 12:51

0 Answers0