2

I am trying to fetch the messages from kafka topic and seeing that if I set the auto.commit.reset policy to "earliest" all the messages are processed properly. But if set to "latest", the first message goes missing and the rest of the messages are processed properly. Can anyone help if I am missing something here?

Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39
  • Does this answer your question? [How to check if Kafka Consumer is ready](https://stackoverflow.com/questions/48071988/how-to-check-if-kafka-consumer-is-ready) – laffuste Jun 30 '21 at 07:23

1 Answers1

0

Assuming the group you are using has not committed any offsets, when setting auto.commit.reset to latest the consumer will only receive new messages, so I'm not exactly sure what you mean by "first message".

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • I am sending a stream of message by auto incrementing the message ID's(starting from 1).The message looks something like this {id: $id_no(this is auto incremented, "message": "This is a sample message")}. On sending a stream of 25 messages, I am able to see outputs from message with ID no 2 onwards. But the message with ID no 1 is not processed and not shown in the output. – Nagireddy Hanisha Dec 19 '17 at 10:23
  • 1
    Is the consumer already connected and ready to consume when you start the producer ? – Mickael Maison Dec 19 '17 at 10:45
  • I am sure that consumer is ready because when I set the auto.commit.reset is set to earliest, the first message is also processed. But not sure what is happening in the background when set to latest only which is causing the problem. – Nagireddy Hanisha Dec 19 '17 at 10:48
  • 1
    Don't assume that the consumer is ready when you use auto.commit.reset latest, consumer group needs to balance first before it is able to receive messages. Launch consumer and wait until group is balanced (you can check logs) before producing. – Luciano Afranllie Dec 19 '17 at 15:39
  • 1
    Okay. Can you guide how do I ensure and wait until consumer balancing is done (or) a workaround for this ? – Nagireddy Hanisha Dec 20 '17 at 09:38
  • Have you come up with any solution to this issue? I am experiencing something similar and this might be useful. – Cristian Ramon-Cortes May 10 '19 at 09:57