2

We have the situation where we have Kafka messages of the same type, which are slightly in the wrong order. For example the Kafka topic has the following messages, each with a creation timestamp:

  1. Message X, 2021-04-26T19:16:16.010
  2. Message Y, 2021-04-26T19:16:16.030
  3. Message Z, 2021-04-26T19:16:16.020

Because of the way, the messages were produced, Message Z is in third position, but based on the creation timestamp should be in second position (before Message Y). The wrong order is only for messages e.g. within a timeframe of 1 second.

We have the requirement, that the Kafka messages are consumed exactly in the order of their timestamps.

One possible solution would be, to consume the messages, cache them shortly in some kind of persistent storage (long enough to gather all messages within above mentioned timeframe) and push them into another Kafka topic in the right order.

But I wonder, if there would be a "Kafka way" of doing this. One without a persistent storage. Could this re-ordering of messages be done with Kafka or Kafka Streams?

stefitz
  • 446
  • 6
  • 10
  • I think a similar question is answered [here.](https://stackoverflow.com/questions/43939534/apache-kafka-order-windowed-messages-based-on-their-value/44345374#44345374) – Sanju Thomas Apr 27 '21 at 01:17
  • Why are they in the wrong order? – PragmaticProgrammer Apr 27 '21 at 02:13
  • 1
    @PragmaticProgrammer I expected this question ;-) The messages come from other system components with variable processing speeds. The timestamp is set, when these components receive an input. We cannot really influence that behaviour, thus we have to "fix" the order. – stefitz Apr 27 '21 at 06:03
  • @SanjuThomas Thanks for the link. I searched quite a bit and did not find this answer - or any other relevant answer. At first sight it looks like a similar implementation to what we came up with, just "wrapped" in a processor. Will definitely look into this solution. – stefitz Apr 27 '21 at 06:10

0 Answers0