0

Why does the offset value in actual topic is different than the offset value in __consumer_offset for the same topic? PFB the offset positions along with the commands used.

__consumer_offsets_13
=====================================
[root@node1 __consumer_offsets-13]# /opt/kafka_2.11-0.10.1.1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files ./00000000000000000000.log | tail -n 5
_offset: 41368_ position: 4190035 CreateTime: 1532888732120 isvalid: true payloadsize: 28 magic: 1 compresscodec: NoCompressionCodec crc: 1789813648 keysize: 43 key: my_consumer_groupmy_topic payload: Ad�I��d�p-�

my_topic
=====================================
[root@node1 __consumer_offsets-13]# /opt/kafka_2.11-0.10.1.1/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --deep-iteration --print-data-log --files ../my_topic-0/00000000000000000000.log | tail -n 5
offset: 2080 position: 315620 CreateTime: 1532891670673 isvalid: true payloadsize: 118 magic: 1 compresscodec: NoCompressionCodec crc: 744326405 payload: {message_content_1}
_offset: 2081_ position: 315772 CreateTime: 1532891670673 isvalid: true payloadsize: 118 magic: 1 compresscodec: NoCompressionCodec crc: 2573656188 payload: {message_content_2}

What am I missing?

Divs
  • 1,578
  • 2
  • 24
  • 51

1 Answers1

1

The kafka.tools.DumpLogSegments tool prints the content of the logs.

The offset you see when running it against __consumer_offsets is the offset of the message that contains the consumer group offset. It is not the offset of the consumer group!

The actual offset of the consumer group is contained in the payload of that message. In your output, it displays as payload: Ad�I��d�p-� because it needs to be decoded.

See Kafka how to read from __consumer_offsets topic if you want to print the content of __consumer_offsets.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Thanks a ton! When I use the command you directed me to, it gives `[console-consumer-63470,__consumer_offsets,13]::[OffsetMetadata[41371,NO_METADATA],CommitTime 1532889777047,ExpirationTime 1532976177047] `, But `_consumer_offsets_13` is the partition for a certain consumer group and it contains offset value for _all_ the topics it touches. How do I figure out if the latest offset for a particular topic (`my-topic`) is also the latest offset position for 'my-topic' in`__consumer_offset-13`(since I figured out the this partition already) – Divs Jul 30 '18 at 05:49
  • In Other words, how can I decode the `payload` present and get the `actual offset`? – Divs Jul 30 '18 at 06:15
  • any advice to figure out the above question? – Divs Aug 03 '18 at 07:00
  • That's a different question but the best way to retrieve the committed offset is to use the `kafka-consumer-groups.sh` tool. For example: `kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group my-group` – Mickael Maison Aug 03 '18 at 08:40
  • It doesn't work with the spring-boot client that I currently have and the officially documented way didn't help and I had to stop by @SO. :-) So if there is any other way that can help me get the payload of the message stored in the consumer_offsets, that would help me indeed. – Divs Aug 03 '18 at 11:42