Prometheus queries: How to give a default label when it is missing?

Question

I'm trying to graph the temperature of my servers with Prometheus's hwmon and Grafana.

Relevant for this are 2 time series that prometheus-node-exporter offers:

• node_hwmon_temp_celsius which has the actual temperatures; it has labels like:
  • {chip="platform_coretemp_0",sensor="temp1"}
• node_hwmon_sensor_label which is a helper time series for temperature sensors that have a name (the Prometheus label called label):
  • {chip="platform_coretemp_0",sensor="temp1",label="core_0"}

On https://github.com/prometheus/node_exporter/issues/631 it is explained that:

labels are not available for all sensors. If all you sensors have a label, you can do something like this to join them:

node_hwmon_temp_celsius{...} * ignoring(label) group_left(label) node_hwmon_sensor_label

But some of my sensors don't have a label. The above PromQL query does not help in that case; group_left(label) returns no results in that case.

Instead, I would like to write a query for the temperatures that has the label label present, always, and defaults it to unknown-sensor-name if the label label is missing.

How can I do this with PromQL?

Answer 1

I finally found a solution (not sure if it's optimal):

(
    # For all sensors that have a name (label "label"), join them with `node_hwmon_sensor_label` to get that name.
    (node_hwmon_temp_celsius * ignoring(label) group_left(label) node_hwmon_sensor_label)
  or
    # For all sensors that do NOT a name (label "label") in `node_hwmon_sensor_label`, assign them `label="unknown-sensor-name"`.
    # `label_replace()` only adds the new label, it does not remove the old one.
    (label_replace((node_hwmon_temp_celsius unless ignoring(label) node_hwmon_sensor_label), "label", "unknown-sensor-name", "", ".*"))
)

How I arrived there:

• Finding all time series that do NOT have the label label is essentially the equivalent of an OUTER JOIN from SQL; googling that for PromQL I found question Is there a way to do a "left outer join" like query in PromQL? that explains that this is unless in PromQL.
• Now we need to add the constant label="unknown-sensor-name". We can do that with label_replace(), which, perhaps confusingly, only really adds a new label, it does not remove the old one (as described here). We can leave the last 2 arguments as "" since we only add a constant string, so we can match any existing label.
• Finally, we can union-join this with the "if all you sensors have a label" series using or.

Done.

---

If desired, we can also join it up with the machine's hostname by adding:

* on(instance) group_left(nodename) node_uname_info

That is described in https://blog.ruanbekker.com/cheatsheets/prometheus/ or Relabel instance to hostname in Prometheus.

That way, in Grafana, we can use {{nodename}} {{chip}} {{label}} for the Legend of the graph.

---

Example in Grafana (showing instance instead of nodename):

Answer 2

Just wrap the recommended query into label_replace() function:

label_replace(
  node_hwmon_temp_celsius
    * ignoring(label) group_left(label)
  node_hwmon_sensor_label,
  "label",
  "unknown-sensor-name",
  "label",
  ""
)

The label_replace() searches for missing label values and then replaces them with the label="unknown-sensor-name". It doesn't change already existing label values, since they do not match the provided empty regular expression in the last argument to label_replace(). Prometheus automatically anchors the regexp to the start and the end of the string, e.g. in reality it uses ^$ regexp, which matches only an empty string.

Answer 3

You can take advantage of the fact that label_replace allows you to supply a regex match as a requirement for replacement, and replace only if it matches against an empty string (i.e. no label present):

label_replace(
  <your metric>,
  <label>,
  <default value>,
  <label>,
  "^$"
)

Or in your case specifically:

label_replace(
  <your metric>,
  "label",
  "unknown-sensor-name",
  "label",
  "^$"
)

Where ^ is the special character used in Regexes to denote the beginning of a string, and $ is the same for the end of a string.