4

We started using spring-integration-kafka in a project, but couldn't find any documentation on suggested approach for error handling on the consumer side.
Ideally we'd want a back-off retry policy without blocking threads, so, it seems stateless retry advice is not an option; however, I can't find an example using stateful retry advice with Kafka.
Any suggestion on how should we handle error retries with spring-integration-kafka in general? Thanks.

heyu
  • 185
  • 1
  • 13
  • Even stateful retry blocks the thread - the only difference is whether the exception is thrown back to the container or not. It doesn't make much sense to use stateful retry with Kafka because there is no concept of competing consumers. – Gary Russell Mar 21 '15 at 19:12
  • Thanks for the answer. How do you suggest to do the retry then? for example, if I'm using a queue, after 3 times non-backoff retry, I can throw it back into the queue, is there anything similar in Kafka, like, after retries, if still fails, do not commit the position? – heyu Mar 21 '15 at 20:21

2 Answers2

2

Please, share a config on the matter and point out where would you like to use a retry advice.

Typically any inbound-channel-adapter puts its messages to some channel with a subscriber like <service-activator> which can be configured with RequestHandlerRetryAdvice.

From other side the <poller> on the <int-kafka:inbound-channel-adapter> can be configured with StatefulRetryOperationsInterceptor as you wish.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • Based on Gary's answer above, it seems like it's not the matter of how to config stateful or stateless advice, but more of how should I handle retry failures with Kafka. Thanks for the hints anyways. – heyu Mar 21 '15 at 20:23
  • You can't achieve that with `Hight-Level Consumer` (``), because it commits offset automatically without our intervention. For the `` (`Simple Consumer`) we have an open [JIRA](https://jira.spring.io/browse/INTEXT-137) ticket, which will be addressed in the nearest future. – Artem Bilan Mar 22 '15 at 08:28
0

Since your requirement is to not block the thread during retry, you could write the failed message to another topic and have some background process consume from this "dead-letter-queue" and re-publish to the original topic some time later.

With rabbitmq this can be done automatically with appropriate configuration of a dead letter exchange, and a dead letter queue with message time to live but I don't think kafka has anything similar so you'll have to roll your own.

Community
  • 1
  • 1
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Yeah, I don't know how Kafka can do this either, I did use DLQ with ActiveMQ, will simulate one myself for Kafka. Thanks. – heyu Mar 22 '15 at 15:43