101

I need to find out a way to ask Kafka for a list of topics. I know I can do that using the kafka-topics.sh script included in the bin\ directory. Once I have this list, I need all the consumers per topic. I could not find a script in that directory, nor a class in the kafka-consumer-api library that allows me to do it.

The reason behind this is that I need to figure out the difference between the topic's offset and the consumers' offsets.

Is there a way to achieve this? Or do I need to implement this functionality in each of my consumers?

Tavo
  • 3,087
  • 5
  • 29
  • 44
  • 3
    Please consider changing the accepted answer given that things have changed and zookeeper is not used in newer versions of Kafka. – Gray Mar 01 '19 at 15:15

9 Answers9

168

Use kafka-consumer-groups.sh

For example

bin/kafka-consumer-groups.sh  --list --bootstrap-server localhost:9092

bin/kafka-consumer-groups.sh --describe --group mygroup --bootstrap-server localhost:9092
Basanth Roy
  • 6,272
  • 5
  • 25
  • 25
  • 7
    we should use --zookeeper not --bootstrap-server – jack AKA karthik Feb 03 '17 at 09:31
  • 1
    but 0.9.0.0 is not an ancient version of kafka :) – jack AKA karthik Feb 06 '17 at 04:55
  • 19
    bin/kafka-consumer-groups.sh --list --zookeeper localhost:2181 Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API). bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092 Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). – Raja Krishnan Jan 25 '18 at 00:12
  • 4
    Consumer group list command lists all the consumers groups in the cluster, but is there a way to show consumer groups for specific topic? – user3366706 Jun 01 '18 at 22:32
  • 1
    @user3366706 `kafka-consumer-groups --topic` only applies when providing `--reset-offsets`, so there would have to be a different flag to create an API call for "groups for a specific topic" – OneCricketeer Oct 29 '19 at 22:34
  • Can you please provide a java implementation of the same? – Abhishek Gharai Oct 31 '19 at 13:32
  • When I execute the list function, I'm getting following error "Error: Executing consumer group command failed due to Request METADATA failed on brokers List" – c0der512 Jan 16 '20 at 18:28
20

you can use this for 0.9.0.0. version kafka

./kafka-consumer-groups.sh --list --zookeeper hostname:potnumber

to view the groups you have created. This will display all the consumer group names.

 ./kafka-consumer-groups.sh --describe --zookeeper hostname:potnumber  --describe  --group consumer_group_name

To view the details 

GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
jack AKA karthik
  • 885
  • 3
  • 15
  • 30
19

I realize that this question is nearly 4 years old now. Much has changed in Kafka since then. This is mentioned above, but only in small print, so I write this for users who stumble over this question as late as I did.

  1. Offsets by default are now stored in a Kafka Topic (not in Zookeeper any more), see Offsets stored in Zookeeper or Kafka?
  2. There's a kafka-consumer-groups utility which returns all the information, including the offset of the topic and partition, of the consumer, and even the lag (Remark: When you ask for the topic's offset, I assume that you mean the offsets of the partitions of the topic). In my Kafka 2.0 test cluster: 
kafka-consumer-groups --bootstrap-server kafka:9092 --describe
    --group console-consumer-69763 Consumer group 'console-consumer-69763' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
pytest          0          5               6               1               -               -               -
``
ynux
  • 1,280
  • 14
  • 21
15

All the consumers per topic

(Replace --zookeeper with --bootstrap-server to get groups stored by newer Kafka clients)

Get all consumers-per-topic as a table of topictabconsumer:

for t in `kafka-consumer-groups.sh --zookeeper <HOST>:2181 --list 2>/dev/null`; do
    echo $t | xargs -I {} sh -c "kafka-consumer-groups.sh --zookeeper <HOST>:2181 --describe --group {} 2>/dev/null | grep ^{} | awk '{print \$2\"\t\"\$1}' "
done > topic-consumer.txt

Make this pairs unique:

cat topic-consumer.txt | sort -u > topic-consumer-u.txt

Get the desired one:

less topic-consumer-u.txt | grep -i <TOPIC>
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
egor7
  • 4,678
  • 7
  • 31
  • 55
  • 2
    I think this is the closest solution to the asked question. This should be the accepted answer. Sometimes you don't know the consumer groups and that's why you'd need this. The only caveat to that solution is that it does not filter out topics in a consumer group where there is no active consumers. Thank you ! – Guillaume Jobin Jan 17 '20 at 18:02
11

I do not see it mentioned here, but a command that i use often and that helps me to have a bird's eye view on all groups, topics, partitions, offsets, lags, consumers, etc

kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --all-groups

A sample would look like this:

GROUP TOPIC PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG CONSUMER-ID HOST CLIENT-ID
Group Topic 2          7               7               0   <SOME-ID>   XXXX <SOME-ID>
:
:

The most important column is the LAG, where for a healthy platform, ideally it should be 0(or nearer to 0 or a low number for high throughput) - at all times. So make sure you monitor it!!! ;-).

P.S:
An interesting article on how you can monitor the lag can be found here.

jumping_monkey
  • 5,941
  • 2
  • 43
  • 58
4

Kafka stores all the information in zookeeper. You can see all the topic related information under brokers->topics. If you wish to get all the topics programmatically you can do that using Zookeeper API.

It is explained in detail in below links Tutorialspoint, Zookeeper Programmer guide

Rohlik
  • 1,286
  • 19
  • 28
Prasad
  • 616
  • 7
  • 11
  • 8
    This is actually incorrect in the newer versions of Kafka. Zookeeper is no longer used to store offsets. The offsets are stored in Kafka itself in specially named topics – Basanth Roy Jun 17 '18 at 22:00
  • Agree with Basanth. But anyhow, providing an answer to to stackoverflow should contain enough question-related and useful information to provide a minimal value. – branko terzic Nov 29 '19 at 09:50
3

High level consumers are registered into Zookeeper, so you can fetch a list from ZK, similarly to the way kafka-topics.sh fetches the list of topics. I don't think there's a way to collect all consumers; any application sending in a few consume requests is actually a "consumer", and you cannot tell whether they are done already.

On the consumer side, there's a JMX metric exposed to monitor the lag. Also, there is Burrow for lag monitoring.

Gray
  • 115,027
  • 24
  • 293
  • 354
sandris
  • 1,363
  • 13
  • 27
2

You can also use kafkactl for this:

# get all consumer groups (output as yaml)
kafkactl get consumer-groups -o yaml

# get only consumer groups assigned to a single topic (output as table)
kafkactl get consumer-groups --topic topic-a

Sample output (e.g. as yaml):

name: my-group
protocoltype: consumer
topics:
 - topic-a
 - topic-b
 - topic-c

Disclaimer: I am contributor to this project

D-rk
  • 5,513
  • 1
  • 37
  • 55
0

And another important point, while running below command, if you see consumer_ID is empty it means its inactive.

kafka-consumer-groups --bootstrap-server kafka:9092 --describe  --group promotion1          

GROUP               TOPIC                       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
promotion1          PromotionBusiness 0          75554           77040           1486            -               -               -         -               -               -
Tyler2P
  • 2,324
  • 26
  • 22
  • 31
Jaydeep Sharma
  • 1
  • 1