How can I group labels in a Prometheus query?

Question

If I have a metric with the following labels:

my_metric{group="group a"}  100
my_metric{group="group b"}  100
my_metric{group="group c"}  100
my_metric{group="misc group a"}  1
my_metric{group="misc group b"}  2
my_metric{group="misc group c"}  1
my_metric{group="misc group d"}  1

Is it possible to do a query or even a label_replace that combines the 'misc' groups together?

(I realize that the metric cardinality needs to be improved, and I've updated the app to fix it. However it left me with this question for if I wanted to fix the metrics via a query later)

score 57 · Answer 1 · answered May 14 '20 at 12:06

57

It's even easier

sum by (group) (my_metric)

answered May 14 '20 at 12:06

uamanager

1,128
8
11

2

Does this solve the second part of the original question of grouping the 'misc' labels? – checketts May 14 '20 at 17:14
1

You can also combine them if you require like `sum(my_metric{group=~"group.*"}) by (group)` – shailendra1118 Aug 24 '20 at 10:21
3

For this query, if I write an alert like `sum by (group) (my_metric) > 0`, how can I access the `group` name? Something like `{{group}}`? – exAres Nov 03 '20 at 04:39

score 55 · Accepted Answer · answered Jul 18 '17 at 10:02

55

Yes, you can you use label replace to group all the misc together:

sum by (new_group) (
  label_replace(
    label_replace(my_metric, "new_group", "$1", "group", ".+"),
    "new_group", "misc", "group", "misc group.+"
  )
)

The inner label_replace copies all values from group into new_group, the outer overwrites those which match "misc group.+" with "misc", and we then sum by the "new_group" label. The reason for using a new label is the series would no longer be unique if we just overwrote the "group" label, and the sum wouldn't work.

answered Jul 18 '17 at 10:02

tom.wilkie

2,846
20
16

1

Isn't it possible to archieve the same result by only doing: `sum by (misc) ( label_replace(my_metrics, "misc", "group", "misc group.+") )` – Hawky4s Mar 09 '18 at 08:31
For grouping the labels based on regex, refer to the accepted answer from this post: https://stackoverflow.com/questions/66669688/group-labels-in-a-prometheus-query-based-on-regex – Hidayat Rzayev Mar 30 '21 at 12:17

score 16 · Answer 3 · answered Jul 18 '17 at 02:57

16

You can use a regex query:

my_metric{group=~"misc group.+"}

That will give you everything where group starts with "misc group".

answered Jul 18 '17 at 02:57

Oliver

11,857
2
36
42

4

This will match all that is misc group.* but not group by. (e.g. all independent metrics that match the regex above will show as a separate metric not all as one metric) – cgseller Dec 28 '21 at 15:15

How can I group labels in a Prometheus query?

3 Answers3

Linked