1

I prefer confluent-kafka-go driver. It's fast but it needs some workaround. The most tricky part is commiting messages. For the better performance we shouldn't commit every message, it's better to do commit periodically. But to make this approach robust we should handle rebalance events.

I have my implementation of such approach. but, how you guys do it?

Or nobody deals with it?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Yura
  • 969
  • 14
  • 33
  • I don't think "rebalancing" is the problem here. Rather duplicate processing if you don't commit, and re-consume the same data **after the rebalance**, yes? – OneCricketeer Feb 03 '23 at 08:32
  • Not exactly. We don't need to commit after every message is processed. Commit may be done after some timeout or messages amount. One corner case that should be taken into account too – if consumer gets partition unassign event, it should commit offsets before that to avoid reconsuming messages when it will be assigned again. Consumer's graceful shutdown is another thing that should be implemented. This is one of possible commit strategies. My question is – do people write it every time? I have my package that does all these things. But I'm curious to compare it with others implementations. – Yura Feb 03 '23 at 10:23
  • 1
    IMO, yes people rewrite this each time before refactoring into a common framework. Because everyone's use-cases are different, and sometimes event consumption is not impacted by rebalancing. As one example, KafkaJS gives `eachMessageBatch` function (similar to Java polling up to 500 records by default), in addition to `eachMessage`, where you can finely commit one-by-one. If you allow for auto-commits, then its already not "every message". – OneCricketeer Feb 03 '23 at 20:26
  • That's interesting! Processing correct commits and shutdowns needs particular attention. And that's interesting that this boilerplate code is not something that people share as "here's how I did this". Usually there are shared packages for such copy-pasted code. Anyway, thanks for you answer and opinion! – Yura Feb 06 '23 at 10:36

0 Answers0