0

Currently, I have setup instances in our cluster to emit metrics over Prometheus, including a static metric using a textfile collector, which looks like this:

prod_server{id="12345"} 1

These metrics help identify the instance with IDs that we use throughout our infrastructure, rather than their IP addresses. Currently, I have an alert setup in Grafana Alerts that uses the following query to get memory usage per instance:

100 - (sum by (instance) (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

This results in Grafana retrieving memory usage percentage per instance, and labelling them by their IP addresses, such as instance=10.10.0.1, instance=10.10.0.2 and so on.

My question is, is there a way to modify these labels/legends such that instead of the IP addresses, the custom textfile collector metric is used to label the instances instead?

Thank you!

frozensaberz
  • 31
  • 5
  • In your exact case: `(100 - (sum by (instance) (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100) * on(instance) group_left(id) prod_server` – markalex May 16 '23 at 21:22
  • 1
    A better way would be to use [relabel_configs](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) to either add a new label with human-readable value, or modify instance value. – anemyte May 17 '23 at 06:17
  • @anemyte, I agree that relabeling a valid alternative. But it's not always a **better** way. For example, if you have dynamic instances and auto-discovery of targets, relabeling is not very obvious or simple to perform. – markalex May 17 '23 at 06:23

0 Answers0