How to execute multiple queries in one call in Prometheus

Question

I'm running prometheus inside kubernetes cluster.

I need to send queries to Prometheus every minute, to gather information of many metrics from many containers. There are too match queries, so I must combine them.

I know how I can ask Prometheus for one metric information on multiple containers: my_metric{container_name=~"frontend|backend|db"} , but I haven't found a way to ask Prometheus for multiple metric information in one query.

I'm looking for the equivalent to 'union' in sql queries.

score 38 · Accepted Answer · answered Nov 21 '17 at 14:49

38

I found here this solution: {__name__=~"metricA|metricB|metricC",container_name=~"frontend|backend|db"}.

answered Nov 21 '17 at 14:49

roie

665
1
6
14

2

The article you link to specifically says that using regex on `__name__` is an antipattern. – Jørgen May 12 '20 at 07:21
2

https://prometheus.io/docs/prometheus/latest/querying/basics/ states no qualms against this construct. – fche Nov 01 '20 at 20:37
3

The linked article says that its example regex with wildcards "is an anti-pattern, as you're meant to know your metric names a-priori." I think the spirit of this question and answer is that you _do_ know the exact names of the metrics you want (that is, you're not using any wild cards in your regex) you just want more than one at a time. To me, this answer doesn't seem to fit into the anti-pattern they're worried about. – mactyr Dec 13 '21 at 20:14

score 6 · Answer 2 · answered Nov 21 '17 at 08:42

6

You can use the or operator, however this does not generalise as it ignores metric names. I'd suggest making multiple queries to the API.

answered Nov 21 '17 at 08:42

brian-brazil

31,678
6
93
86

1

Thanks, easiest and most elegant solution for quickly debugging a query in Prometheus' query tool. – Antoine Viallon Nov 16 '22 at 09:51

valyala · Answer 3 · 2023-04-10T20:03:40.590

Prometheus provides or operator, which can be used for combining results from multiple queries in a single API call. For example, the following query returns results for q1, q2 and q3:

q1 or q2 or q3

The or operator in PromQL has a quirk though - if some time series returned by q1, q2 and q3 have the same set of labels (excluding metric name aka __name__ label), then only the first encountered time series is returned. For example, the following query returns only process_cpu_seconds_total time series, and doesn't return process_resident_memory_bytes time series, since they have the same set of labels except of metric name:

process_cpu_seconds_total or process_resident_memory_bytes

This can be fixed by copying the metric name into another label. For example, the following query copies the metric name into metric_name label, so the or result contains time series from both process_cpu_seconds_total and process_resident_memory_bytes sides:

label_join(process_cpu_seconds_total, "metric_name", "", "__name__")
  or
label_join(process_resident_memory_bytes, "metric_name", "", "__name__")

P.S. There is a simpler solution for obtaining results from multiple queries in one call if using an alternative Prometheus-like monitoring system I work on - VictoriaMetrics - it provides union function for this task. For example, the following query returns time series for both process_cpu_seconds_total and process_resident_memory_bytes queries:

union(
  process_cpu_seconds_total,
  process_resident_memory_bytes,
)

How to execute multiple queries in one call in Prometheus

3 Answers3

Linked