Kafka consumer goes into an unending loop

Question

I am using Kafka queue to hold some objects that is to be retreived by consumer app and perform some operation on it.

Problem : If the processing by consumer takes more than ~2 hrs kafka seems to give back the same object again and again

Code :

  private static Queue queue = new LinkedList();
  while (true) {
            ConsumerRecords<String, String> records = consumer.poll(Long.MAX_VALUE);
            for (ConsumerRecord<String, String> record : records) {
                System.out.println("\n\n\n Kafka has :[" + record.offset());
                queue.add(record.value());
            }
            System.out.println("\n\n\n Kafka has :[" + records.count());
            if (queue != null) {
                maintainQueue();
            }
        }

score 1 · Accepted Answer · edited May 23 '17 at 12:00

1

I am using kafka version 0.10.1.0

We have resolved this issue by updating ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG to "false" and adding consumer.commitSync(); after consumer.poll(Long.MAX_VALUE);

Reference :

Kafka - How to commit offset after every message using High-Level consumer?

http://www.slideshare.net/jjkoshy/offset-management-in-kafka

edited May 23 '17 at 12:00

Community

1
1

answered Feb 03 '17 at 05:10

ericdemo07

461
8
16

Kafka consumer goes into an unending loop

1 Answers1