how to add label to table from Prometheus SNMP exporter using 2 values from another table

Question

I am using SNMP exporter to provide metrics from an APC lab monitoring device (using PowerNet MIB).

The relevant part of the Prometheus configuration looks as follows:

  - job_name: labtemp_snmp
    static_configs:
      - targets:
        - 10.163.123.87
    scrape_interval: 15s
    metrics_path: /snmp
    params:
      module: [labtemp]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9116

The Prometheus SNMP exporter configuration was generated using:

modules:
  labtemp:
    walk:
      - memSensorsStatusTable
    version: 1

When querying by hand ('http://localhost:9116/snmp?target=10.163.123.87&module=labtemp'), the output contains these tables:

# HELP memSensorsStatusSensorLocation The location of the sensor. - 1.3.6.1.4.1.318.1.1.10.4.2.3.1.4
# TYPE memSensorsStatusSensorLocation gauge
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="0",memSensorsStatusSensorLocation="Cold Aisle C1",memSensorsStatusSensorNumber="1"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="0",memSensorsStatusSensorLocation="Hot Aisle C2",memSensorsStatusSensorNumber="2"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="1",memSensorsStatusSensorLocation="Cold Aisle C12",memSensorsStatusSensorNumber="2"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="1",memSensorsStatusSensorLocation="Hot Aisle B8",memSensorsStatusSensorNumber="1"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="2",memSensorsStatusSensorLocation="Cold Aisle A5",memSensorsStatusSensorNumber="1"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="2",memSensorsStatusSensorLocation="Hot Aisle B2",memSensorsStatusSensorNumber="2"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="3",memSensorsStatusSensorLocation="Cold Aisle C9",memSensorsStatusSensorNumber="3"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="3",memSensorsStatusSensorLocation="Hot Aisle C12",memSensorsStatusSensorNumber="6"} 1
memSensorsStatusSensorLocation{memSensorsStatusModuleNumber="3",memSensorsStatusSensorLocation="Hot Aisle C7",memSensorsStatusSensorNumber="5"} 1
# HELP memSensorsTemperature The sensor's current temperature reading - 1.3.6.1.4.1.318.1.1.10.4.2.3.1.5
# TYPE memSensorsTemperature gauge
memSensorsTemperature{memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="1"} 26
memSensorsTemperature{memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="2"} 33
memSensorsTemperature{memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="1"} 30
memSensorsTemperature{memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="2"} 20
memSensorsTemperature{memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="1"} 22
memSensorsTemperature{memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="2"} 30
memSensorsTemperature{memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="3"} 20
memSensorsTemperature{memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="5"} 31
memSensorsTemperature{memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="6"} 29

Now, I'd like to assign the memSensorsStatusSensorLocation values to the memSensorsTemperature table using the memSensorsStatusSensorLocation table, similarly how this is done e.g. with network interface indexes, so that the entries look like this:

memSensorsTemperature{memSensorsStatusSensorLocation="Cold Aisle C1",memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="1"} 26
...

For the record, the relevant part of the MIB looks like this:

memSensorsStatusTable OBJECT-TYPE
   SYNTAX SEQUENCE OF MemSensorsStatusEntry
   ACCESS not-accessible
   STATUS mandatory
   DESCRIPTION
       "Allows for getting sensor status information."
   ::= { memSensors 3 }
   
memSensorsStatusEntry OBJECT-TYPE
   SYNTAX MemSensorsStatusEntry
   ACCESS not-accessible
   STATUS mandatory
   DESCRIPTION
      "The unit to get data from."
   INDEX { memSensorsStatusModuleNumber, memSensorsStatusSensorNumber }
   ::= { memSensorsStatusTable 1 }

Clearly, the 2 numbers serve as an index.

There is also another table memSensorsConfigTable that contains memSensorsLocation:

PowerNet-MIB::memSensorsLocation.0.1 = STRING: "Cold Aisle C1"
PowerNet-MIB::memSensorsLocation.0.2 = STRING: "Hot Aisle C2"
PowerNet-MIB::memSensorsLocation.1.1 = STRING: "Hot Aisle B8"
PowerNet-MIB::memSensorsLocation.1.2 = STRING: "Cold Aisle C12"
PowerNet-MIB::memSensorsLocation.2.1 = STRING: "Cold Aisle A5"
PowerNet-MIB::memSensorsLocation.2.2 = STRING: "Hot Aisle B2"
PowerNet-MIB::memSensorsLocation.3.3 = STRING: "Cold Aisle C9"
PowerNet-MIB::memSensorsLocation.3.5 = STRING: "Hot Aisle C7"
PowerNet-MIB::memSensorsLocation.3.6 = STRING: "Hot Aisle C12"

Anyone can provide a way/hint how to achieve this ?

It is not clear to me whether this should/could be done in Prometheus SNMP exporter configuration or Prometheus configuration.

I'm not sure that I understand this question fully, even with your own answer. But wouldn't ["joining" metrics](https://stackoverflow.com/questions/44461801) do what you want here? — markalex, May 22 '23 at 19:25

Vlad · Answer 1 · 2023-05-15T20:03:35.223

Use the memSensorsConfigTable and perform double-index lookup. The generator configuration looks like this:

modules:
  labtemp:
    walk:
      - memSensorsStatusTable
      - memSensorsConfigTable
    version: 1
    timeout: 60s
    lookups:
      - source_indexes: [memSensorsStatusModuleNumber, memSensorsStatusSensorNumber]
        lookup: memSensorsLocation

With that, the manual query contains the extra label:

# HELP memSensorsTemperature The sensor's current temperature reading - 1.3.6.1.4.1.318.1.1.10.4.2.3.1.5
# TYPE memSensorsTemperature gauge
memSensorsTemperature{memSensorsLocation="Cold Aisle A5",memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="1"} 22
memSensorsTemperature{memSensorsLocation="Cold Aisle C1",memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="1"} 26
memSensorsTemperature{memSensorsLocation="Cold Aisle C12",memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="2"} 20
memSensorsTemperature{memSensorsLocation="Cold Aisle C9",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="3"} 20
memSensorsTemperature{memSensorsLocation="Hot Aisle B2",memSensorsStatusModuleNumber="2",memSensorsStatusSensorNumber="2"} 30
memSensorsTemperature{memSensorsLocation="Hot Aisle B8",memSensorsStatusModuleNumber="1",memSensorsStatusSensorNumber="1"} 30
memSensorsTemperature{memSensorsLocation="Hot Aisle C12",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="6"} 29
memSensorsTemperature{memSensorsLocation="Hot Aisle C2",memSensorsStatusModuleNumber="0",memSensorsStatusSensorNumber="2"} 33
memSensorsTemperature{memSensorsLocation="Hot Aisle C7",memSensorsStatusModuleNumber="3",memSensorsStatusSensorNumber="5"} 31

The trouble is that the additional table causes the SNMP scraping to take much longer than previously (than just with the memSensorsStatusTable). Hence, the timeout had to be raised, as well as scraping interval in Prometheus configuration:

  - job_name: labtemp_snmp
    static_configs:
      - targets:
        - 10.163.123.87
    scrape_interval: 60s
    scrape_timeout: 60s
    metrics_path: /snmp
    params:
      module: [labtemp]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9116

how to add label to table from Prometheus SNMP exporter using 2 values from another table

1 Answers1