1

I am using kafka-dump-log tool to see the content of the __consumer_offsets topic. Can someone explain what does that content mean exactly ( offset, createtime, isvalid, keysize, valuesize, compresscodec, producerId, producerEpoch, isTransactional, headerKeys, key, offset, payload ) ?

[root@localhost kafka_2.11-2.0.0]# bin/kafka-dump-log.sh --files data/kafka/__consumer_offsets-32/00000000000000036360.log --value-decoder-class "kafka.serializer.StringDecoder" --offsets-decoder
    Dumping data/kafka/__consumer_offsets-32/00000000000000036360.log
    Starting offset: 36360
    offset: 36360 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 0 isTransactional: false headerKeys: [] key: offset::b0147:test:48 payload: 35
    offset: 36361 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 1 isTransactional: false headerKeys: [] key: offset::b0147:test:64 payload: 34
    offset: 36362 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 2 isTransactional: false headerKeys: [] key: offset::b0147:test:14 payload: 34
    ...
    offset: 36460 position: 5497 CreateTime: 1623678503139 isvalid: true keysize: 9 valuesize: 24 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: {"metadata":"b0147"} payload: {"protocolType":"consumer","protocol":null,"generationId":2,"assignment":"{}"}
    offset: 36461 position: 5598 CreateTime: 1623678554656 isvalid: true keysize: 19 valuesize: -1 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: []
    ...
xRobot
  • 25,579
  • 69
  • 184
  • 304

1 Answers1

5

The fields you see are not specific to __consumer_offsets, they are just the metadata about each message/batch in the log and you will get them for all topics, see https://kafka.apache.org/documentation/#recordbatch

If you want to explore the content of __consumer_offsets you need to know that it contains 2 types of records:

  • OffsetMetadata: this is information about the offsets committed by groups
  • GroupMetadata: this is information about members of consumer groups

To show OffsetMetadata, you can use:

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets

To show GroupMetadata, you can use:

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets
Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Good answer. Will complement with the usage of that particular topic: https://stackoverflow.com/questions/39529511/what-is-the-use-of-consumer-offsets-and-schema-topics-in-kafka#:~:text=__consumer_offsets%20is%20used%20to,only%20latest%20offsets%20information%20available. – Ignacio Acuña Frías Jun 14 '21 at 16:08
  • I tried your last command but I get an empty result. Why? # bin/kafka-console-consumer.sh --formatter "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter" --bootstrap-server localhost:9092 --topic __consumer_offsets ^CProcessed a total of 200 messages – xRobot Jun 15 '21 at 07:58
  • Do you have any existing consumer groups? Is the other command working? – Mickael Maison Jun 15 '21 at 08:43
  • why is there `32` suffix in `__consumer_offsets-32`. Important for me to understand this... if someone knows please answer. – samshers Dec 06 '22 at 07:36
  • does it mean - partition 32.. hoping so. – samshers Dec 06 '22 at 08:29
  • yes after the dash there is the partition number – Mickael Maison Dec 06 '22 at 18:13