0

We try to feed metrics from worldwide distributed devices into Prometheus. To decouple the devices from the services (to avoid too many direct communications), there is a message broker and a Kafka bridge.

+----------------------------------+
|   Device 1                       |
| +---------------+                |
| | Application I |--+             |
| +---------------+   \            |
|                      |           |
| +----------------+   |  +------+ |              +------+     +----------+       +--------------+
| | Application II |---+->| MQTT |-|---metrics--->| MQTT |---->|  Kafka   |------>| Prometheus   |
| +----------------+   |  +------+ |              +------+     +----------+       +--------------+
|                      |           |
| +-----------------+  /           |
| | Application III |-+            |
| +-----------------+              |
+----------------------------------+

We struggle to figure out how to feed Prometheus from the Kafka bridge more or less directly. We could - of course - instantiate translation services per Application (Service) on each device, but this feels being overkill.

We didn't find any suggestion on https://prometheus.io/docs/instrumenting/exporters/ - but I think we ask the wrong questions.

Our first application is a collectd to measure some common system metrics. This is a reasonable data source since everyone knows what should be seen in Prometheus then.

The communication

+------+                +------+     +----------+
| MQTT |----Internet--->| MQTT |---->|  Kafka   |
+------+                +------+     +----------+

cannot reasonable opened up for particular messages, since each message shall receive Kafka.

Sno
  • 179
  • 1
  • 7
  • The question is maybe sibling to https://stackoverflow.com/questions/61825022/how-to-export-data-from-kafka-to-prometheus or https://stackoverflow.com/questions/50517152/report-prometheus-metrics-via-kafka. – Sno Apr 01 '22 at 14:52
  • 2
    Prometheus pulls its data from exporters. The exporters don't push, they expose metrics at an HTTP endpoint. JMX exporter for Kafka will export Kafka's metrics, and so, I think what you need is a custom Prometheus exporter that pulls data from Kafka and translates your application metrics into something Prometheus understands. – ashu Apr 01 '22 at 14:53

1 Answers1

0

There is only a remote write integration for Kafka from Prometheus.

If you intend to collect metrics from external resources, you wouldn't write to Kafka at all, rather add some exporter (ex. collectd exporter) to the applications themselves, then setup scrape targets against them since Prometheus is pull-based.

If you really wanted to use Kafka, you would need to write a consumer first, then use PushGateway to push data; there is nothing provided by Prometheus project for this, that I am aware of. Or you can basically rewrite (or pay Confluent for) that Kafka Connect project mentioned here.

One alternative, install a StatsD server as a relay - https://github.com/prometheus/statsd_exporter#relaying-from-statsd

Push to that, rather than Kafka. (Or again, consume from Kafka, and push to the statsd server)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Since the collectd (and other applications) are „remote“ it is not reasonable to write directly from collectd into Prometheus. – Sno Apr 03 '22 at 07:37
  • Not sure I understand the question. Again, Prometheus would be **pulling** collectd-exporter – OneCricketeer Apr 03 '22 at 12:56
  • No, prometheus cannot pull the collectd-exporter. This is prohibited. – Sno Apr 04 '22 at 13:06
  • By who / why? What about statsd? Regardless, like I said, there's nothing that can push to Prometheus except for PushGateway. Or you need a different exporter. For example, there's this project I found on Github. https://github.com/bendikwa/mqtt_exporter – OneCricketeer Apr 04 '22 at 14:26
  • Prohibition is enforced by authentication architecture. We check the statsd relay as well as the Confluent Kafka Connect project and an own service. I finally tried to understand the limitations around and got that the transport layer is not optimized for prometheus architecture. – Sno Apr 05 '22 at 09:49
  • 1
    Cortex has a push endpoint that might work better (that's what we run), but I do think InfluxDB or TimescaleDB might be better as there's better Kafka tooling that can be used with those – OneCricketeer Apr 05 '22 at 12:58