15

When the consumer does not receives a message for 5 mins (default value of max.poll.interval.ms 300000ms) the consumer comes to a halt without exiting the program. The consumer process hangs and does not consume any more messages.

The following error message gets logged

MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 255ms (adjust max.poll.interval.ms for long-running message processing): leaving group

I see that ErrMaxPollExceeded is defined here in confluent-kafka-go but unable to find where it is getting raised.

If any such error is raised, why does the program not exit ?

  • Kafka v1.1.0
  • librdkafka v1.0.0
  • confluent-kafka-go (master)

Config used for kafka.Consumer

{
    "bootstrap.servers":    "private.kafka.host",
    "group.id":             "foo.bar",
    "auto.offset.reset":    "earliest",
    "enable.auto.commit":   false,
}
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
esquarer
  • 373
  • 1
  • 4
  • 12
  • Are you calling Poll() or ReadMessage() with a timeout longer than the max.poll.interval.ms? If so, you're probably hitting an issue in librdkafka which is fixed in librdkafka v1.1.0. – Edenhill Jun 24 '19 at 20:18
  • 1
    @Edenhill No, I'm calling `Poll()` with a timeout of 100 ms – esquarer Jun 25 '19 at 07:37
  • Hi @esquarer did you find solution, please let me know am facing same the issue. – Sawood Oct 24 '20 at 09:04

1 Answers1

4

It looks like this is a reported issue to the confluent-kafka-go client: https://github.com/confluentinc/confluent-kafka-go/issues/344

It is marked as resolved by this change which should have been released in librdkafka version 1.1.0: https://github.com/edenhill/librdkafka/commit/80e9b1ee0c0e477c02f18b130c4302240711a88d

It looks like that change should also have been released in confluent-kafka-go version 1.1.0: https://github.com/confluentinc/confluent-kafka-go/tree/v1.1.0

Some comments in the issue still report that this is an issue experiences on versions >1.1.0.

A commenter suggests a work around:

Using v1.5.2. Also calling ReadMessage(-1) in an infinite loop, and not seeing rejoining after consumer leaving group, worked around it by setting timeout to be less than max.poll.interval.ms instead of -1, but wondering why it's not rejoining as expected.

https://github.com/confluentinc/confluent-kafka-go/issues/344#issuecomment-745014864

dolan
  • 1,716
  • 11
  • 22