1

Suppose I'm debugging an issue involving a single specific message that was produced and consumed. And I want to know when this message was produced and when it was consumed. What are my options for getting this info?

I guess when I construct a message I could include within it the current time. And when my consumer gets a message, it could write out a log entry.

But suppose I have many producer and consumer classes and none of the code is doing these things. Is there something already existing in kafka that could support finding out this info about a specific message without having to touch the implementation of these producers and consumers, something like the __consumer_offsets topic?

user3735178
  • 129
  • 10

2 Answers2

2

Kafka has built-in timestamp support for messages sent and this timestamp can be accessed via timestamp method of ConsumerRecord (link)

It can be configured with broker config (log.message.timestamp.type) or topic level config (message.timestamp.type). Its default value is CreateTime. You can also set this as LogAppendTime.

CreateTime: Timestamp is assigned when producer record is created (before sending).

LogAppendTime: broker will override the timestamp with its current local time and append the message to the log.

IMHO for consume timestamp your only option is to get currentTime of the system after message process is finished.

For more information about timestamp you can check this.

H.Ç.T
  • 3,335
  • 1
  • 18
  • 37
1

When it comes to consumption, there's no explicit way of specifying when message was consumed (also bear in mind that a single message can be consumed multiple times e.g. by consumers in different consumer groups).

However there are a few possible ways to track it on your own:

  • register the timestamps after the record has been received (after the .poll(...) call returns),
  • if using consumer groups, monitor consumer group's offset or see the value in __consumer_offsets (that would require you to deserialize internal format - see this answer for details (bear in mind, the timestamps of records in this topic correspond to consumer's commit timestamps, so they need to commit often enough to provide correct granurality,
  • log compaction + custom implementation: send the message with the same key and value that marks the consumption timestamp (however the message might still be re-read before the compaction happens).
Adam Kotwasinski
  • 4,377
  • 3
  • 17
  • 40