Kafka consumer not able to read from all available partitions

Question

We are noticing a weird issue with one of our prod topics(6 partitions) where our consumer (dotnet core, only 1 instance) is only able to read from 3 partitions (0,1,3). This is obviously impacting application behavior as the consumer is missing the messages from the other 3 partitions (2,4,5). We were able to validate there is nothing wrong with topic configuration. Suspecting an offset commit issue due to message expiry, we deleted all the messages from the topic (by changing the retention period to a small number) but that did not resolve the issue. We have tried several restarts of the consumer app and every time the partition assignment remains the same (0,1,3) which makes us believe there is something wrong with the other partitions. We are contemplating the thought of creating a new topic. Thought, I will post it here to check if we are missing something. Any inputs is appreciated. Thanks.

Answer 1

1. To start with, check and confirm all the partitions are receiving messages from producers
2. Once it is confirmed, check the partition assignment strategy during the consumer start and confirm the consumer client is assigned with all partitions of the topic. if it is not possible using consumer client itself, kafka-consumer-groups.sh utility can be used
3. Ensure that there are no other consumer client(s) with same group_id running and consuming messages from the topic

List all consumer groups using kafka-consumer-groups.sh

kafka-consumer-groups.sh --list --bootstrap-server <kafka-broker>:<port>

Above command will list all the consumer groups for all topics managed by the cluster.

After narrowing down to the consumer group you are interested in, list the consumers part of the consumer group using below command.

kafka-consumer-groups.sh --bootstrap-server <kafka-broker>:<port> --describe --group <your-consumer-group>

kafka-consumer-groups.sh script returns all the information about a topic partition including

• current offset - current max offset of the consumed messages of the partition for this consumer instance
• log end offset - offset of the latest message in the partition
• lag - difference between consumed offset and latest offset
• client ID - ID to differentiate individual clients in the same consumer group

Example 1: current offset was 4, lag is 2, and client ID is empty - means consumer client is no longer registered and processing any messages. Also note the message saying there are no active members in the consumer group.

kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --describe --group my-group

Consumer group 'my-group' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-group        my-topic        0          4               6               2               -               -               -

Example 2: current offset was 4, lag is 2 and client ID is not empty - means consumer client is not processing messages at the moment. However, it processed some messages earlier.

kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --describe --group my-group

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                      HOST            CLIENT-ID
my-group        my-topic        0          4               6               2               test-client-31ca0f5d-958e-4a30-b028-e3d975b41fd6 /0.0.0.0    test-client

Example 3: LOG-END-OFFSET is 0. Hence, no messages produced to this topic so far.

kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --describe --group my-group

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                      HOST            CLIENT-ID
my-group        test1           0          0               0               0               test-client-08763d84-c4be-49f8-a75f-286c1968f6f9 /0.0.0.0    test-client

Example 4: CURRENT-OFFSET is 0. but LOG-END-OFFSET is 2 and LAG is also 2 - means there are messages in this topic partition. However, it is not being consumed.

kafka-consumer-groups.sh --bootstrap-server <broker-host>:9092 --describe --group my-group_new_4

Consumer group 'my-group_new_4' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
my-group_new_4  test1           0          0               2               2               -               -               -

This will help you to figure out the questions asked at the beginning of this answer, and isolate/fix the issue.

Answer 2

Are you using a Group Consumer Client? If so there is a possibility that another consumer (a different application) is consuming from this topic using the same "group_id". In that case, those other 3 partitions (2, 4, 5) could be assigned to that other consumer client.