1

My setup:

  • Prometheus server
  • 2 node-exporters
  • Grafana

docker-compose.yml

version: '2'

services:

  prometheus_srv:
    build: ./prom_serv
    image: prom/prometheus
    container_name: prometheus_server
    hostname: prometheus_server

  prometheus_node:
    image: prom/node-exporter
    container_name: prom_node_exporter
    hostname: prom_node_exporter
    depends_on:
      - prometheus_srv

  prometheus_node2:
    image: prom/node-exporter
    container_name: prom_node_exporter2
    hostname: prom_node_exporter2
    depends_on:
      - prometheus_node

  grafana:
    image: grafana/grafana
    container_name: grafana_server
    hostname: grafana_server
    depends_on:
      - prometheus_node2

Dockerfile for Prometheus server:

FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/

prometheus.yml

# Load and evaluate rules in this file eve
scrape_configs:
  # Scrape Prometheus itself
  - job_name: 'prometheus'
    scrape_interval: 10s
    scrape_timeout: 10s
    static_configs:
      - targets: ['localhost:9090']

  # Scrape the Node Exporter
  - job_name: 'node'
    scrape_interval: 10s
    static_configs:
      - targets: ['prom_node_exporter:9100']

  # Scrape the Node Exporter2
  - job_name: 'node2'
    scrape_interval: 10s
    static_configs:
      - targets: ['prom_node_exporter2:9100']

After adding the Prometheus datasources to Grafana, I add a new dashboard with 2 CPU-usage graphs, one for each node exporter:

100 - (avg by (instance) (irate(node_cpu{job="node",mode="idle"}[5m])) * 100)
100 - (avg by (instance) (irate(node_cpu{job="node2",mode="idle"}[5m])) * 100)

and try to generate a CPU spike to the first node-exporter like this:

docker container exec -it prom_node_exporter sh
/ # dd if=/dev/zero of=/dev/null

What I finally see is that the two graphs look quite similar:

enter image description here

I suppose that the CPU usage should be much higher to the container the command was used. What goes wrong here? Any suggestions?

tgogos
  • 23,218
  • 20
  • 96
  • 128

1 Answers1

5

I assume you're running all your containers on the same host. The node exporter will export the host information - and therefore both node exporters will show the same information about the one host.

To collection information about CPU usage of containers, please use cAdvisor: It will allow you to collect CPU usage per container.

You should run one cAdvisor per host to collect information about the containers on the host, and one node exporter per host to collect information about the host.

ahus1
  • 5,782
  • 24
  • 42
  • Thanks! You are right, I am running the containers on the same host. I will check cAdvisor then... – tgogos Jun 29 '17 at 19:26
  • 1
    I also found this: [Prometheus - how to monitor other docker containers](https://stackoverflow.com/questions/43999394/prometheus-how-to-monitor-other-docker-containers/44000227) which helped me a lot... – tgogos Jun 30 '17 at 12:15