1

I am trying to understand some fundamental Kafka concepts so that I can properly monitor the progress of my KafkaStreams based application.

Specifically for debugging purposes I need to be able to have my application re-consume a whole topic. For that I used the reset tool.

After executing the script looking into the Kafka Manager for some inputs topics I see that the Consumer Offset has decreased and the Lag has increased (which makes sense). Although the Consumer Offset is not going to zero. I am trying to interpret that but I haven't found a concrete explanation of what the Consumer Offset and Logsize in Kafka Manager are referring to.

To make it fit what I see I assume that the Logsize is the total amount of messages placed into the topic since it's beginning but not necessarily the amount of messages currently in the topic. As some may have been thrown away due their age exceeding the retention period. Am I right?

If not, then what is the explanation behind the fact that after running the reset tool for some input topics I observe that the Consumer Offset is equal to the Logsize (and not zero) and Lag is zero?

LetsPlayYahtzee
  • 7,161
  • 12
  • 41
  • 65

1 Answers1

2

I am not familiar with yahoo-kafka-manager, however, you can also use bin/kafka-consumer-groups.sh (a tool shipped with Kafka itself). There LOG-END-OFFSET means what you describe. From a naming perspective it's unclear to me if Logsize is the same as "log end offset" or the difference between highest and lowest offset in a partition.

After executing the script looking into the Kafka Manager for some inputs topics I see that the Consumer Offset has decreased and the Lag has increased.

This makes sense -- as "lag" is difference of "log end offset" and "committed offset" the lag should be increased after resetting you applications. However, I am not sure why committed consumer group offset is not zero (can you very what you observe using bin/kafka-consumer-group.sh -- maybe yahoo-kafka-manager report something different).

Update: however the tool will not set the offset to zero but to "beginning of log". (The docs are not correct.)

Also note, that auto.offset.reset strategy might tick in after you reset your applications and restart it ([committed] offset zero might not be valid if log got truncated). Could this explain the behavior you observe?

This blog post might also help to understand further details: https://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • by writing data to some test topic I validated that my assumption about what Logsize means is correct. But I think that the `auto.offeset.reset` does not have to do with what am I seeing. I think that the reset tool itself picks the latest available offset and doesn't reset the offset to zero if zero is not available. I checked that by setting the value to both latest and earliest and seeing nothing changing. If the reset tool would have put the offset to zero the `auto.offset.reset` would have behaved differently in each case. Can it be the case that the reset tool is updated that way? – LetsPlayYahtzee May 09 '17 at 11:23
  • 1
    Hmmm... Not sure. Btw: the docs are not completely correct. The tool "seeksToBeginning" of input topics (cf https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/StreamsResetter.java#L223) So `auto.offset.reset` should not trigger when you start Streams app after resetting (what does make sense -- the docs are just not correct :( -- I will open an PR to fix the docs for the next release) – Matthias J. Sax May 10 '17 at 04:37