Read latest message from kafka - segmentio/kafka-go

Question

I'm using segmentio/kafka-go client to read messages from a topic. I'm unable to find.. how to start reading from last/new message.

Everytime I start the code, it starts reading from beginning offset in that partition.

Answer 1

What you need to know about consuming messages from Kafka is that each consumer client is part of a Consumer Group. Kafka stores the already processed offset for each Consumer Group at Topic-Partition level in an internal Kafka topic called __consumer_offsets. This enables a consumer of a Consumer Group to continue consumption from where it left off after a re-start.

In your case it means you need to set the Consumer Group (in the KafkaConsumer API it is the configuration "group.id") and keep it constant. Only then you will be able to continue reading from the latest/new est message and not start from beginning after a re-start.