2

Given a metrics with various labels (i.e., same metric name with different labels), is it possible to aggregate these at scrape time into a single value, e.g., by summing all the values?

I know this is possible at query time, but I'm asking about scrap time as idea is to reduce the number/cardinality of metrics stored in prometheus.

For example, given the following metric output:

some_metric{server="server-0"} 30  1395066363000
some_metric{server="server-0"} 70  1395066363000
some_metric{server="server-0"} 100 1395066363000

I'd like to drop the server label and import it as if it was:

some_metric 200 1395066363000
BeeOnRope
  • 60,350
  • 16
  • 207
  • 386
  • AFAIK, it's not possible at scrape time. You can remove a label there, but you cannot perform `sum()` on colliding metrics. It is either remove the label from the exporter, or accept unnecessary cardinality. – anemyte Mar 25 '22 at 07:31

2 Answers2

1

You can use recording rules for aggregating metrics after they are scraped. Note that the original metrics are indexed and saved before the recording rule is applied to them. So the original metrics may require additional RAM, CPU and disk space.

If you do not want storing the original metrics, then you can try stream aggregation in VictoriaMetrics (this is a Prometheus-like project I work on). It allows aggregating scraped metrics in stream mode, e.g. without saving them to disk. For example, the following -stremAggr.config config instructs removing the host label from the scraped metrics with the name some_metric and storing only the sum of these metrics every 30 seconds:

- match: some_metric
  interval: 30s
  without: [host]
  outputs: [sum]
valyala
  • 11,669
  • 1
  • 59
  • 62
0

I think you can use relabel_configs or metrics_relabel_configs and labeldrop.

Be careful that you don't cause a collision in your resulting metrics though (two metrics previously differing by their labelsets could collide).

There's an explanation of the difference between them here

DazWilkin
  • 32,823
  • 5
  • 47
  • 88
  • The problem is that I _want_ to cause "collisions" as per my example: all three values should be treated as the same metric and aggregated in some way. – BeeOnRope Mar 25 '22 at 18:26