0

Is there a way to decompose the messages so that only the latest message is consumed?

I tried to save the messages in a list, but it didn't really work out

var consumer = new Consumer(new ConsumerOptions(topic, router));

foreach (var message in consumer.Consume())
{
    Console.WriteLine(Encoding.UTF8.GetString(message.Value));
}

Output should be: 1, 2, 3, 4

Output is: 1, 1, 2, 1, 2, 3, 1, 2, 3, 4

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
point
  • 290
  • 3
  • 12

2 Answers2

2

You can use something called Log Compaction which will "truncate" messages with the same key. So when you send messages with the same key you will get only the last message/value for that key. This feature is enabled by default. When you send the messages 1, 1, 2, 1, 2, 3, 1, 2, 3, 4 and you want to read them as 1, 2, 3, 4 you give the messages the same keys, which should be overwritten by each other (so all the 1 messages get the same key, all the 2 messages get the same key, ...).

Progman
  • 16,827
  • 6
  • 33
  • 48
0

As a matter of fact, the only way to do what you want is to have a topic with single partition and setting max.poll.records to one.

Otherwise, there is no way to to it, because the last message does not make sense. Any input message can be pushed into different partitions and you have some last messages relative to the number of partitions your topic has.

Amin Heydari Alashti
  • 881
  • 1
  • 9
  • 20
  • You can still get the message with the highest offset *per partition* – OneCricketeer Apr 21 '19 at 16:15
  • @cricket_007 I described completely in my answer. If you do so what you mentioned, you will have a set of biggest offsets relative to the number of partitions in your topic not the biggest one. But, your solution can be true and it needs an additional step; the biggest offset among the biggest ones in each partition should be found. – Amin Heydari Alashti Apr 21 '19 at 17:01
  • Well, it wasn't exactly mentioned how many partitions in the question there are. Plus, it's not guaranteed that 1-4 are exactly in their matching partitions – OneCricketeer Apr 21 '19 at 20:03