PromQL. How to add label value from different metric

Question

If I have two metrics:

kube_pod_container_status_restarts_total {..., pod="my_pod_name_42", ...}

and

container_memory_usage_bytes {..., pod_name="my_pod_name_42", cluster_name="megatron", ...}

And I want to set up alert for restart metrics, but containing cluster_name label. Can I somehow aggregate label value from different metrics.

For example alert:

increase(kube_pod_container_status_restarts_total{namespace="42"}[1h])>4    

need to somehow add megatron here ^, for alertmanager to be able to use this label when sending notification

Answer 1

You need to use label_replace() function in order to convert pod_name label into pod label before using group_left:

(increase(kube_pod_container_status_restarts_total{namespace="42"}[1h]) > 4)
+ on (pod) group_left(cluster_name)
(0 * label_replace(
  container_memory_usage_bytes{namespace="42"},
  "pod", "$1", "pod_name", "(.+)"
))

As you can see the label_replace() function isn't trivial to use for labels' renaming. The better approach is to use label_move() function from MetricsQL. Then the label_replace(container_memory_usage_bytes{namespace="42"}, "pod", "$1", "pod_name", "(.+)") is substituted with more clear label_move(container_memory_usage_bytes{namespace="42"}, "pod_name", "pod")

Answer 2

Possible duplicate of How can I 'join' two metrics in a Prometheus query? .

One improvement could be to force the metric you are joining to not affect yours by doing something like this (addition of a metric forced to be 0):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)