How to monitor messages rate in Kafka topics?

Question

How can I get alerted when there is a message rate in some topic higher or lower than usual?

I've edited your question because asking for tools is considered off-topic. [help] But basically, you need to enable JMX on the brokers and (Java) clients, then you need a tool to export JMX values into a aggregate system, such as Prometheus, as mentioned, but there are still other options — OneCricketeer, Jan 14 '19 at 00:29

score 6 · Accepted Answer · edited Jan 14 '19 at 00:32

Kafka emits tons of metrics that allow to monitor its usage and health. This includes metrics for incoming message rate, see the Monitoring section in the docs.

Your use case is typically achieved with a tool that handles metrics and is able to fire alerts based on user defined queries.

For example, Prometheus handles that very well. You need to set it up so metrics from Kafka are forwarded to Prometheus (using https://github.com/prometheus/jmx_exporter). As it's a very common use case, you'll find hundreds of articles online that describe how to set that up. The JMX exporter has sample configs for the Kafka JMX patterns, as well.

Once the metrics are in Prometheus you can aggregate them across brokers and set up Alerts.

How to monitor messages rate in Kafka topics?

1 Answers1