We have a Kafka consumer that consumes an internal topic, processes the message and sends it to an external partner over rest:
Topic ---> Consumer ---> Partner HTTP endpoint
We frequently get 404 responses for some of the messages we send and want to implement a retry mechanism. Due to the circumstances, we want to retry every second week (14 days period).
Most of the information I found on retries in Kafka are rather short-lived and not periodic. Like having a few retries topics which you iterate through:
retry_14d_topic' — for retry in 14 days
retry_28d_topic' — for retry in 28 days
retry_42d_topic' — for retry in 42 days
Every time the periodic retry runs, we want to send all the data in the topic. One way would be to have a consumer that runs in a cron-job, with a script that resets the topic offset every time before it runs. However, this does not feel very Kafka-like and quite akward. What's a good design in this case? Should we avoid trying to do this with Kafka in the first place?
