2

I am reading from Kafka topic, and I commit the read offsets.

Can I somehow, using the same group.id and without seeking on a particular partition as suggested here, re-read the entire topic from beginning to end?

using Kafka 1.1 my pseudo code:

var config = new Dictionary<string, object>
{
 { "group.id", "NCC1"},
 ...
 { "enable.auto.commit", "false"},
 { "default.topic.config", new Dictionary<string, object>
 {
  { "auto.offset.reset", "earliest" } 
 }
 }
};      

using (var consumer = new Consumer<string, string>(config, new StringDeserializer(Encoding.UTF8), new StringDeserializer(Encoding.UTF8)))
{
  consumer.Subscribe(new string[] { topic });
  consumer.OnMessage += (_, msg) =>
  {
    // DO SOMETHING
    onsumer.CommitAsync(msg);
  };
Greg Bala
  • 3,563
  • 7
  • 33
  • 43

2 Answers2

2

You can use kafka-consumer-group to reset offsets for the existing group ids.

kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> 
--reset-offsets --to-earliest --execute

It will reset the offset for the given topic and group.id to the earliest.

You can see more options there:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling

Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101
1

Kafka allows you to reset the offset for a particular topic and group id. This possibility of replay is actually one of the selling points of Kafka.

You can do it using the kafka-streams-application-reset commandline tool as documented here.

An alternative way is to use kafka-consumer-groups command line tool as described in this blog. The alternative way might be interesting if you are using Windows since the equivalent batch file for kafka-streams-application-reset.sh is not provided.

Obviously, you can move your offset backward as long as the events are still in the topic. Events already removed from the topic cannot be replayed.

EDIT

I think your question is quite similar to this one

HL'REB
  • 838
  • 11
  • 17