15

I'm new at using Kafka and I have one question. Can I delete only ONE message from a topic if I know the topic, the offset and the partition? And if not is there any alternative?

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
Ander99
  • 163
  • 1
  • 1
  • 4

1 Answers1

21

It is not possible to remove a single message from a Kafka topic, even though you know its partition and offset.

Keep in mind, that Kafka is not a key/value store but a topic is rather an append-only(!) log that represents a stream of data.

If you are looking for alternatives to remove a single message you may

  1. Have your consumer clients ignore that message

  2. Enable log compaction and send a tompstone message

  3. Write a simple job (KafkaStreams) to consume the data, filter out that one message and produce all messages to a new topic.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • Considering our event stream context, there's a fourth option that aligns more with the event-sourcing paradigm. Instead of attempting to tombstone an erroneous message, you could consider appending a new `Failed` event to the end of the topic. This approach involves shifting the pointer within your consumer. The `Failed` event could encompass a reference to the original unprocessed event along with a record of the attempted actions. This technique allows the consumer's pointer to progress while retaining the means to retry the problematic event :) – Aaron Newton Aug 10 '23 at 07:11
  • P.S. the above solution would only be appropriate if you wanted to advance the consumer pointer and retry at the end of the current stream, but hopefully that's clear from context. – Aaron Newton Aug 10 '23 at 07:19