1

In my kafka java-project I want to delete the messages as soon as all interested consumers have received the new message. After some research I have found some old stackoverflow questions: here, one more and here. After reading all these, I've got some questions.

As far as i could understand, I really should rely on retention either by time or by space. However, the answers are old so maybe something changed? Is there any other way to really ensure that messages are deleted right after all the currently connected consumers have read the message? In this case I would need to check whether or not all consumers have read the message. Would I need a consumer-group for that?

Thank you in advance.

Aksim Elnik
  • 425
  • 6
  • 27
  • Why do you want to delete messages after consuming it? is it to avoid duplicate message delivery or any other reason? – Govinda Sakhare Jun 25 '20 at 14:22
  • Why do you want to do that? Kafka is designed to be a distributed commit log. The behaviour you want is a publish-subscribe system, like a JMS server, RabbitMQ or maybe amazon SNS. – GeertPt Jun 25 '20 at 14:27
  • @GoviS The thing is, that I need the consumers to really process the message. So I want to delete the message once all consumers have actually received and processed it. Currently the processing time as well as the message-size can vary like a lot so i really cannot put time or size constraints as for retention. – Aksim Elnik Jun 25 '20 at 14:53
  • @GreyFairer the answer above – Aksim Elnik Jun 25 '20 at 14:55
  • I can let consumers commit after processing, but would that help me? – Aksim Elnik Jun 25 '20 at 14:56
  • Just to know if consumers have processed all messages, they need to commit what they consumed, and you can check their offsets. – GeertPt Jun 25 '20 at 15:09
  • @GreyFairer So theoretically I could let the adminClient delete the message once all consumers commited? – Aksim Elnik Jun 25 '20 at 16:15
  • @AksimElnik theoretically, yes, but it's not necessary, as the answer below points out. – GeertPt Jun 26 '20 at 09:10

1 Answers1

0

You don't need worry about when the messages will be deleted or compacted to achieve your goal

As far as your consumer is concerned, if the consumers commit the offset of every message they consumed, those processed messages, regardless of how long they will stay in the topic, are dead in the view of your consumers.

Note:

An administrative user can go and reset your consumer to read gain from the beginning including those messages your consumers moved on from. But that is a manual admin operation.

Kavithakaran Kanapathippillai
  • 7,883
  • 1
  • 11
  • 29
  • Maybe interesting to add: If you deploy a new consumer with a new consumer group, you can also choose to either start with offset 0, to receive old messages again, or start from the current offset, and only receive new messages. – GeertPt Jun 26 '20 at 09:15