0

Use case

I wrote a prometheus exporter for Kafka Consumer Group offsets (https://github.com/cloudworkz/kafka-minion) which works by consuming the internal __consumer_offsets topic. Key and value for all messages inside of this topic are binary and I am able to decode them. When any group offsets expire Kafka is supposed to create a tombstone for this key (which consists of group id, topic name and partition id).

The problem

This approach works very well so far, however Kafka apparently does not produce a tombstone for some expired offsets / groups sometimes. I consumed the offsets topic with this command:

kafka-console-consumer.sh --from-beginning --topic __consumer_offsets zookeeper.connect=${ZOO_HOST} --bootstrap-server="${KAFKA_HOST}"  --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter

This produces formatted log lines which contain the offset commits along with their group names, partitionID and commit timestamp. It will also print the tombstones (NULL value). There are some consumer group offsets which still have no tombstones yet, even though their offsets have been commited 6 months ago (offset retention is set to 2w). When I list consumer groups from Kafka (using the other shell scripts), Kafka does not list this consumer group either - so internally Kafka is apparently aware that these offsets are expired.

The question

Why does Kafka sometimes does not produce tombstones for expired offsets? How does Kafka know these offsets are expired when it apparently does not rely on receiving Tombstones for expired group offsets?

kentor
  • 16,553
  • 20
  • 86
  • 144

1 Answers1

0

Expired records aren't tombstoned: the file segments are simply deleted from disk.

Sounds like you're maintaining state within the application without a TTL on the records you've consumed

You may want to inspect how other Prometheus Lag exporters or Burrow work

You also don't need zookeeper.connect as part of that command

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Can you elloborate your first sentence? I mean this is a compact topic and the messages are indeed still there (as I checked against with the console consumer). How would these messages get deleted if not via a tombstone (produced by the brokers)? The tombstones for the expired offsets are sent by the kafka brokers (and this is generally the case), but sometimes it does not send these tombstones (which I don't understand). – kentor Dec 28 '19 at 11:44
  • Also see: https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management – kentor Dec 28 '19 at 12:09
  • That page hasn't been updated in 6 years, and claims to be obsolete. The property `offsets.retention.minutes` cleans up expired offsets. The topic is not indefinitely compacted and the topic config is actuality `compact,delete` https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist – OneCricketeer Dec 28 '19 at 17:11
  • Since the message is still existent in the topic, the offset can not be deleted even if the cleanup policy is compact & delete (which is not the case for my cluster btw) – kentor Dec 28 '19 at 17:58
  • Correct, and in that case, it's also not tombstoned, so I'm not sure I understand what you're trying to say. I simply answered how the messages are deleted since their not tombstoned. Describing the topic won't show both properties for the offsets topic. You could try setting that property to -1, therefore disabling it – OneCricketeer Dec 28 '19 at 18:03
  • The problem is that they are NOT being deleted because they are not tombstoned. The consumer offsets topic is by default configured to be compacted. Also the Kafka documentation says the brokers are supposed to ALWAYS produce a tombstone once offsets expire. This is also the case for most of the consumer groups, but not for all the groups. – kentor Dec 28 '19 at 19:01
  • It's defaulted to compact *and delete*, as mentioned. Tombstoning isn't the only condition for segment deletion... Can you please link to the documentation you refer? Shown here, segments are clearly being deleted https://stackoverflow.com/questions/39131465/how-does-an-offset-expire-for-an-apache-kafka-consumer-group And if that's not happening for some groups, sounds like you still have an active one – OneCricketeer Dec 28 '19 at 21:16
  • Also, are you sure they're committed a long time ago? I don't see a print.timestamp property on your consumer command – OneCricketeer Dec 28 '19 at 21:22