Backoff strategy for apache kafka consumer

Question

When using kafka component for camel, there is two ways to retry when consuming from kafka :

• in-memory retry, using the generic error handling on the camel route. The problem though is that while retrying, the consumer stops polling the broker, and if the max.poll.interval.ms is reached, Kafka broker considers the consumer as unhealthy, and removes it from the consumer group :

org.apache.kafka.clients.consumer.internals.AbstractCoordinator | [Consumer clientId=consumer-1, groupId=2862121d-ddc9-4111-a96a-41ba376c0143] This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

• polling on each retry using parameter breakOnFirstError. The offset is not updated and we keep polling same message from the broker. The problem is that i cannot find a way to define a backoff policy and retries are re-attempted too often.

Do you know how to define a backoff policy for the second approach ?

Answer 1

I am not familiar with Apache Camel, but if you are able to modify the consumer parameters and polling loop, then the second approach is the right one here, it's the Kafka way for retry - do not commit offset, so the next polling loop iteration would consume that message again.

The further strategy depends on what exactly you need in case of failure handling:

• Do you expect retry to eventually succeed? Then to avoid spamming the same message you can adjust an interval at which consumer polls the messages from Kafka with max.poll.interval.ms config parameter. More details here

• Do you want to retry a certain amount of times then proceed to the next message? In this case, you would need to manually implement a retry counter in a polling loop. Once you reach a certain amount of retries - you would simply move consumer further:

  final TopicPartition topicPartition = new TopicPartition(topic, partition); consumer.seek(topicPartition, consumer.position(topicPartition) + 1);