0

I installed Kafka in a DC/OS cluster using the framework.

I would like to get some metrics to monitor it, I saw that there is this configuration:

"TASKCFG_ALL_KAFKA_METRICS_REPORTERS": "com.airbnb.kafka.kafka08.StatsdMetricsReporter",

I looked around in the code a saw that in server.properties.mustache are some configuration parameters:

external.kafka.statsd.port={{STATSD_UDP_PORT}}
external.kafka.statsd.host={{STATSD_UDP_HOST}}
external.kafka.statsd.reporter.enabled=true

Then I looked in the node where one of the brokers is running and the configuration has this values:

external.kafka.statsd.port=57925
external.kafka.statsd.host=198.51.100.1
external.kafka.statsd.reporter.enabled=true
external.kafka.statsd.tag.enabled=true

In this node, at that port, there is a processes listening with mesos-agent in its name.

How can I see the metrics reported by the Kafka brokers?

Facundo Casco
  • 10,065
  • 8
  • 42
  • 63

2 Answers2

0

Using dcos kafka-configuration files or environment variables (DCOS uses Kafka-Mesos Framework) is currently not possible to open a JMX port and monitor the kafka metrics produced by your cluster as described here: Enable JMX on Kafka Brokers. (Change Request for dcos v1.12)

However DC/OS offers a REST-API for receiving cluster, hosts, containers, and applications specific metrics: e.g. v 1.9 (the one I am currently using) https://docs.mesosphere.com/1.9/metrics/metrics-api/#/

EXAMPLE

If you know the agent_id (dcos-node on which your broker is running) e.g. your agent_id is "ed14928-04d1-4f7e-b544-0a3e9d58645b-S9" then:

  1. https:///system/v1/agent/2ed14928-04d1-4f7e-b544-0a3e9d58645b-S9/metrics/v0/containers -> see which containers a running on that agent and find the one responsible for your kafka-broker. e.g. "a8b0d630-c55a-4e1c-a456-69e9a3ad1c16"

  2. https:///system/v1/agent/2ed14928-04d1-4f7e-b544-0a3e9d58645b-S9/metrics/v0/containers/a8b0d630-c55a-4e1c-a456-69e9a3ad1c16/app -> view kafka-metrics

    {"name":"kafka.server.BrokerTopicMetrics.BytesOutPerSec.1MinuteRate","value":0,"unit":"","timestamp":"2018-01-29T15:33:10Z","tags":{"topic":"XXXX"}},{"name":"kafka.log.Log.NumLogSegments","value":2,"unit":"","timestamp":"2018-01-29T15:32:39Z","tags":{"partition":"1","topic":"XXXXX"}},{"name":"kafka.network.RequestMetrics.ThrottleTimeMs.p98","value":0,"unit":"","timestamp":" ... ....

If you do_not_know the agent_id, you have to go over all slaves (agents) https:///mesos/master/slaves e.g. {slaves...."id":"2ed14928-04d1-4f7e-b544-0a3e9d58645b-S13" ...} and fiter the right one. OR if you have the IP address of your broker-node, you can use dcos cli (you have to login first!)

$ dcos node

to find out the agent-ID.

Valentin
  • 1
  • 1
0

Dcos has provided cli to fetch metrics of node and apps (tasks). This cli uses dcos metrics API's, which Valentin specified above.

$ dcos task metrics details <task-id> [--json]
metrics details
    Print a table of all metrics for the task specified by <task-id>

$ dcos task metrics summary <task-id> [--json]   
metrics summary
    Print a table of key metrics for the task specified by <task-id>

<task-id>
    A full task ID, a partial task ID, or a regular expression.
--json
    Print JSON-formatted list of tasks.

$ dcos node metrics details <mesos-id> [--json]
metrics details
    Print a table of all metrics for the agent node specified by <mesos-id>.

$ dcos node metrics summary <mesos-id> [--json]
metrics summary
    Print CPU, memory and disk metrics for the agent node specified by
    <mesos-id>.

--mesos-id=<mesos-id>
    The agent ID of a node.
--json
    Print JSON-formatted list of nodes.
Parvez Kazi
  • 660
  • 5
  • 13