1

I'm new to kafka and am setting up an ELK stack to collect JMX metrics and log files for analysis and reporting.

Currently, I've setup my logstash to collect kafka mbeans in the following patterns:

    {
            "object_name" : "kafka.*:type=*,name=*,topic=*,partition=*"
    },
    {
            "object_name" : "kafka.*:type=*,name=*,topic=*"
    },
    {
            "object_name" : "kafka.*:type=*,name=*"
    },
    {
            "object_name" : "kafka.*:type=*,client-id=*"
    },
    {
            "object_name" : "kafka.*:type=*,name=*,clientId=*,brokerHost=*,brokerPort=*"
    },
    {
            "object_name" : "kafka.*:type=*,name=*,clientId=*"
    },
    {
            "object_name" : "kafka.*:type=*,broker-id=*,fetcher-id=*"
    },
    {
            "object_name" : "kafka.*:type=*,client-id=*"
    }

The issue that I'm having is that I'm missing all kafka.consumer metrics and when I log into to jconsole on the RMI port, I don't see kafka.consumer mbeans there either.

I'm new to this whole JMX/mbeans stuff and kafka, so am I looking in the right place for kafka.consumer metrics? I currently do have active producers and consumers hitting the cluster.

lightweight
  • 3,227
  • 14
  • 79
  • 142
  • did you end up finding what was the issue here? – eyalzek Jun 07 '17 at 09:24
  • were you able to find kafka.consumer ? – Divyanshu Jimmy Sep 18 '18 at 12:28
  • brother check my repository, I wrote a lot about it, you just need to install it using install.sh, and start all of them, then just type jconsole when everything is up... https://github.com/mirkan1/kafka_monitoring/blob/master/start_consumer.sh#L2 – Raqun Bob Jan 06 '23 at 22:32

1 Answers1

2

The kafka.consumer mbeans are only present on the consumer processes themselves, not the Kafka broker processes. Note that the mbeans won't exist if you use a consumer client library other than the java one (e.g. kafka-python).

This is the official section in the Apache documentation about the mbeans that are available on the broker/consumer/producer, and here is a slightly easier to read version from the Confluent Kafka documentation.

RovingBlade
  • 226
  • 1
  • 5
  • But how can I access those mbeans from my java consumer application? Are they accessible remotely via some JMX host:port through jconsole tool? I found no mention of default JMX port for consumer metrics of java kafka-clients lib. – MeetJoeBlack Jun 26 '18 at 13:21
  • 1
    This might help: https://stackoverflow.com/questions/10331189/how-to-find-the-default-jmx-port-number – RovingBlade Jun 27 '18 at 18:18