0

we have one consumer that consume data from kafka with 5 partitions. the consumer have 5 groupid's.

we noticed that each groupid consume messages from only one partition. in the code we Subscribe to the topic using consumer.subscribe.

as we understand with one consumer each groupid need to consume messages from all partitions.

are we wrong in our assumption? if not what causes the problem?

private void subscribe() {
        consumer.subscribe( Collections.singletonList( topic ) );
    } catch ( IllegalArgumentException  e ){
        LOG.error( "Kafka Failed to subscribe to kafka client ", e );
        e.printStackTrace();
    } catch( IllegalStateException e ){
        LOG.error( "Kafka Failed to subscribe to kafka client ", e );
        e.printStackTrace();
    } catch ( Exception e ){
        LOG.error( "Kafka Failed to subscribe to kafka client, topic: " + topic , e );
    }

}
C.Frank
  • 359
  • 1
  • 3
  • 7
  • Ideally the consumer should read from all partitions in case of unique group.id. Can you share your code? – Nishu Tayal Jan 03 '19 at 20:43
  • Does the topic data has a valid partition key? See: https://stackoverflow.com/questions/37511922/topics-partitions-and-keys – Javier Ramos Jan 03 '19 at 23:52
  • 1
    My guess is, either in run-time all group ids are same, or you don't have 5 partitions. Sharing the code where you assign consumer properties and a describe on the Kafka topic would help. – Bitswazsky Jan 04 '19 at 03:27
  • In a consumer group, when you have multiple consumers, every consumer reads from partitions exclusively. Meaning consumer 1 would read from partition 1 and no other consumer would read from that partition, consumer 2 would read from partition 2 and 3 and other consumers won't read from those partitions. And the consumers that don't read become inactive until some other consumer fail. I hope this is relevant and makes sense to you. – ZainNazirButt Mar 25 '20 at 19:17

0 Answers0