0

We faced the problem few times. When a consumer group is created it doesn't contain all topic partitions. So, the consumer group didn't consume all messages. For example, in a topic with 6 partitions, the partition 1 was missing in the consumer group.

The problem is fixed by recreating the consumer group but we need to avoid this situation in the future. So, when and why a consumer group miss one or many partitions ?

Mohamed Gara
  • 2,665
  • 3
  • 13
  • 19
  • What partition strategy are you using? – Jessica Vasey Jan 23 '23 at 18:01
  • Messages are dispatched on different partitions using message key partitioning and the partition assignment algorithm is RangeAssignor. – Mohamed Gara Jan 23 '23 at 22:18
  • With RangeAssignor class the partitions should be divided by the number of consumers. If it doesn't divide evenly the first few consumers (in the lexicographic order) will get one extra partition. Do you have a static group ID for the consumer? If not, whenever there is a rebalance the consumer group will get a new ID and this can shuffle the partition assignments. – Jessica Vasey Jan 24 '23 at 11:27
  • Do you have a static group ID for the consumer? yes, it is the service name. The problem is not shuffling but losing partitions in consumer group. – Mohamed Gara Jan 30 '23 at 11:24
  • I was wondering if the consumer may be losing partitions during/after the shuffle. Do you see any patterns i.e. did you recently increase the partitions in the topic? or did you notice a rebalance? The top answer on this Stack Overflow post should help you debug a little further! https://stackoverflow.com/questions/70396920/kafka-consumer-not-able-to-read-from-all-available-partitions – Jessica Vasey Jan 30 '23 at 12:37
  • The best answer to this problem is here https://stackoverflow.com/questions/58640511/missing-events-on-previously-empty-partition-when-restarting-kafka-streams-app#58646619 – Mohamed Gara May 23 '23 at 20:24

0 Answers0