Two instances consuming from topic

Question

I have deployed two instances of an application. Both applications runs the same code and consumes from the same topic.

@KafkaListener( offsetReset = OffsetReset.EARLIEST, offsetStrategy = OffsetStrategy.DISABLED )
public class AppConsumer implements ConsumerRebalanceListener, KafkaConsumerAware {

    @Topic("topic")
    public void consumeAppInfo(@KafkaKey String name, @Body @Nullable String someString) {
         ...
    }

}

I have a problem where only one of the applications consumes the message. The topic has only one partition, partition 0, which i believe is default.

I have tried to add group-id and threads to the KafkaListener. This seems to work sometimes and other time not.

@KafkaListener(groupId="myGroup", threads=10)

What is the simplest solution to getting both applications to consume the same message?

Use different group id in each application. – Majid Hajibaba Aug 24 '20 at 15:15 — Majid Hajibaba, Aug 24 '20 at 15:15

score 2 · Answer 1 · answered Aug 24 '20 at 14:22

You could not do the group and just give each application a separate consumer id each consumer consumes all messages (unless they are also assigned to a group).

Groups are used for parallel processing of messages each consumer in a group get assigned to a partition for processing messages.

More info => difference between groupid and consumerid in Kafka consumer

score 1 · Accepted Answer · answered Aug 24 '20 at 14:19

1

In kafka, only one consumer within consumer group is assigned to each partition. If you want to consume the data from the same partition by different applications, you need to specify different consumer groups for each different application / instance.

answered Aug 24 '20 at 14:19

Mikalai Lushchytski

1,563
1
9
18

Two instances consuming from topic

2 Answers2