2

I have installed kubernetes metrics-server https://github.com/kubernetes-sigs/metrics-server It works fine, I can use "kubectl top pods" & "kubectl top nodes", Now I want to scrape the metrics to prometheus outside kubernetes cluster to be shown on Grafana, What I have tried are

  1. Expose the metrics-server service using NodePort but not working, some https issue appear
  2. Install metrics-server-exporter from https://github.com/ghouscht/metrics-server-exporter/ (this one only scrape few metrics, I cannot see it on grafana) and https://github.com/grupozap/metrics-server-exporter/ (this one is not an http application, so I cannot expose as a NodePort Service)

What should I do?

Aditia Rahman
  • 90
  • 2
  • 11

2 Answers2

1

Try these out and let me know.

1.First you should run kube-state-metrics which collects all kubernetes metrics .

2.Using pod annotations on the kube-state-metrics , expose metrics like

prometheus.io/scrape: 'true'
prometheus.io/port: 'port'
prometheus.io/path: '/metrics'
prometheus.io/scheme: 'http'

This should expose your metrics from the pod level . To check this you can exec in to the pod and do a curl request on the port followed by the path

3.Now run a prometheus agent which can scrape metrics from this port and send it to the backend db server

4.Configure prometheus ds in your grafana agent and it should be done

Sunjay Jeffrish
  • 102
  • 3
  • thanks for the info, kube-state-metrics works fine, but should I scrape all nodePort ip from prometheus? or 1 nodePort is enough? where should I put pod annotation like you said? is it on deployment spec? on my Grafana the pods list is show but no data for CPU, Memory, Bandwith usage https://prnt.sc/witosr, I'm using this template https://grafana.com/grafana/dashboards/12120 – Aditia Rahman Jan 08 '21 at 07:46
  • its working now, I don't know some metrics are different on grafana dashboard template, so I need to edit by myself ex: kube_pod_container_resource_requests_memory_bytes{pod="$pod", container=~"$container"} i edited to kube_pod_container_resource_requests{pod="$pod", resource="memory", container=~"$container"} – Aditia Rahman Jan 09 '21 at 03:48
  • I don't understand how this can be accepted answer, when initial question for Prometheus instance outside of Kubernetes cluster? New question is https://stackoverflow.com/questions/70457308/add-kubernetes-scrape-target-to-prometheus-instance-that-is-not-in-kubernetes – Paul Verest Dec 23 '21 at 03:20
0

You can use helm to install kube-prometheus-stack.

It will install Node Exporter, kube-state-metrics and prometheus operator along with other dependencies to allow you to monitor your cluster.

You can install it by using:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://charts.helm.sh/stable
helm repo update
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

Once it is installed you can visit your grafana URL and configure dashboard.

You can add prometheus as data source [Configuration -> Data Source] in grafana (prometheus URL should be set by default) and import kubernetes dashboard to show you summary metrics about containers running on Kubernetes nodes.

kool
  • 3,214
  • 1
  • 10
  • 26
  • thanks for the suggestion, but for know I'm still learning and I prefer installing manualy one by one, so I can understand what happen in my k8s cluster – Aditia Rahman Jan 09 '21 at 03:52