3

As it can be guest from the title, is there a way to get the consumer list on a specific topic in java? Untill now Im able to get the list of topics like this

    final ListTopicsResult listTopicsResult = adminClient.listTopics();
    KafkaFuture<Set<String>> kafkaFuture = listTopicsResult.names();
    Set<String> map = kafkaFuture.get();

but I havent found a way to get the list of consumers on each topic

Ares91
  • 506
  • 2
  • 8
  • 27
  • https://stackoverflow.com/questions/32697999/kafka-consumer-list might work for you? – Stefan May 01 '19 at 14:29
  • @Stefan thx for answering, I have already seen it but nothing useful, there are no info about consumers for topic – Ares91 May 01 '19 at 14:45

3 Answers3

14

I was recently solving the same problem for my kafka client tool. It is not easy, but the only way, which I found from the code is the following:

Properties props = ...//here you put your properties
AdminClient kafkaClient = AdminClient.create(props);

//Here you get all the consumer groups
List<String> groupIds = kafkaClient.listConsumerGroups().all().get().
                       stream().map(s -> s.groupId()).collect(Collectors.toList()); 

//Here you get all the descriptions for the groups
Map<String, ConsumerGroupDescription> groups = kafkaClient.
                                               describeConsumerGroups(groupIds).all().get();
for (final String groupId : groupIds) {
    ConsumerGroupDescription descr = groups.get(groupId);
    //find if any description is connected to the topic with topicName
    Optional<TopicPartition> tp = descr.members().stream().
                                  map(s -> s.assignment().topicPartitions()).
                                  flatMap(coll -> coll.stream()).
                                  filter(s -> s.topic().equals(topicName)).findAny();
            if (tp.isPresent()) {
                //you found the consumer, so collect the group id somewhere
            }
}

This API is available from the version 2.0. There is probably a better way but I was not able to find one. You can also find the code on my bitbucket

Katya Gorshkova
  • 1,483
  • 9
  • 16
4

Consumers aren't tied to a topic. They are tied to consumer groups

To get all the consumers of a topic, you must first list all groups, then filter out your topic within each group

Which would start with AdminClient.listConsumerGroups followed by AdminClient.describeConsumerGroups

That gets you a list of descriptions that contain the "members" where you can find the topics

https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/ConsumerGroupDescription.html

Note: there's external tools that make this much easier like Hortonworks SMM https://docs.hortonworks.com/HDPDocuments/SMM/SMM-1.2.0/monitoring-kafka-clusters/content/smm-monitoring-consumers.html

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
4

I am aware that this topic is a bit old, but I'm currently working on a dashboard that needs to list topics with relevant consumer groups. Katya's answer is ok, but it still won't list TopicPartitions for consumer groups without active members. I came across this solution, hope it helps:

Properties props = ...//here you put your properties
AdminClient kafkaClient = AdminClient.create(props);

Map<TopicPartition, OffsetAndMetadata> offsets = kafkaClient
    .listConsumerGroupOffsets("consumer_group_name")
    .partitionsToOffsetAndMetadata()
    .get();

TopicPartition in offsets map can be parsed for specific topic. This listing can be repeated for whole groupId list from Katya's answer.

AlYosha
  • 189
  • 1
  • 4