Helm, Promethus : Install prometheus with data/default directory on ec2 instance

Question

I am working on prometheus inside a Kubernetes enviornment, where I want to monitor our pods, which are being prepared to send metrics to prometheus directly. I am able to install prometheus via helm install stable/prometheus command, but the prometheus.yml scraping file is inside pods, and is also not persistent if pod restarts.

Since we are still experimenting, the scraping file will go through some iterations before we can say for sure this one works for us. Reason why I am sticking with helm is it also installs other packages like grafana, nodeexpoerter, etc, which are helpful

How can I instruct helm to use a specific data directory present on AWS. Let's say /var/prometheus. If that's not possible, then atleast a custom prometheus.yml, which when updated on server side can be reflected in prometheus pods.

As of now, atleast, I would like to add

        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - action: labeldrop
            regex: '(kubernetes_pod|app_kubernetes_io_instance|app_kubernetes_io_name|instance)'


This in the scraping file. What am I missing? Thank you. :-)

Max Lobur · Answer 1 · 2023-01-13T14:00:52.970

0

The issue here is that Prometheus will not reload the updated config automatically. To address this you have a couple of options:

Write a scrape config in a form of ConfigMap, and use hash annotation trick to force a rolling update (think restart) of the Prometheus pods.
Write a scrape config in a form of ConfigMap, and use a built-in reloader from a stable Prometheus chart to handle updates.
Leverage frameworks like kube-prometheus-stack which generate scrape config basing on Pod/Service labels, and automatically reload the updated config into Prometheus.

edited Jan 13 '23 at 14:00

answered Oct 26 '20 at 09:10

Max Lobur

5,662
22
35

How can I use hostpath with helm install prometheus? – We are Borg Oct 26 '20 at 09:45
So I've been just looking thru stable chart vars and found this https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L660-L665 and this https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L321-L353 Since the reloader is a part of a Prometheus chart already, you can try combining these two sections and get the automatic solution with `hostPath` (part 1 of my answer) – Max Lobur Oct 26 '20 at 09:57
Updated my answer, unfortunately reloader does not cover hostPath based config updates, so both parts of my answer still apply. – Max Lobur Oct 26 '20 at 10:24
A `hostPath` volume refers to a specific host directory on the node where the pod happens to be running; if the pod is deleted and recreated on a different node then the data won't follow it. Beyond some specific cases involving DaemonSets, you don't usually want a `hostPath` volume. – David Maze Jan 02 '23 at 13:58
correct, updated – Max Lobur Jan 13 '23 at 14:01

Helm, Promethus : Install prometheus with data/default directory on ec2 instance

1 Answers1