8

I got an alert while configuring the monitoring module using prometheus/kube-prometheus-stack 25.1.0.

Alert

[FIRING:1] KubeProxyDown - critical
Alert: Target disappeared from Prometheus target discovery. - critical
 Description: KubeProxy has disappeared from Prometheus target discovery.
 Details:
  • alertname: KubeProxyDown
  • prometheus: monitoring/prometheus-kube-prometheus-prometheus
  • severity: critical

I think it is a new default rule in kube-prometheus-stack 25.x.x. It does not exist in prometheus/kube-prometheus-stack 21.x.x.

The same issue happened in the EKS and minikube.

KubeProxyDown Rule

alert: KubeProxyDown
expr: absent(up{job="kube-proxy"}
  == 1)
for: 15m
labels:
  severity: critical
annotations:
  description: KubeProxy has disappeared from Prometheus target discovery.
  runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeproxydown
  summary: Target disappeared from Prometheus target discovery.

How can I resolve this issue?

I would be thankful if anyone could help me

Boney Jacob
  • 111
  • 1
  • 6
  • I don't understand the issue. Is it that the rule was changed between 25 and 21? – aclowkay Dec 27 '21 at 09:03
  • What kubernetes version are you using? Did you check if your `kube-proxy` operator is running? There is a similar issue [here](https://stackoverflow.com/questions/60734799/all-kubernetes-proxy-targets-down-prometheus-operator) – mdobrucki Dec 27 '21 at 09:47
  • Same problem here, running a k3s cluster. Alarm first appeared after upgrading prometheus-stack to version 25.x.x – Timbo Dec 30 '21 at 19:19
  • > What kubernetes version are you using? 1.19 > Did you check if your kube-proxy operator is running? Yes, kube-proxy operator is running – Boney Jacob Jan 04 '22 at 04:25

3 Answers3

5

There was a change in metrics-bind-address in kube-proxy. Following the issues posted here, here and here. I can suggest the following. Change kube-proxy ConfigMap to different value:

$ kubectl edit cm/kube-proxy -n kube-system
## Change from
    metricsBindAddress: 127.0.0.1:10249 ### <--- Too secure
## Change to
    metricsBindAddress: 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system
mdobrucki
  • 462
  • 1
  • 7
  • As @demisx mentioned, in his answer to this ticket, it seems to be a typo in this answer: the configmap is not called `kube-proxy` but `kube-proxy-config`. There you can find the `metricsBindAddress` property mentioned here. – atavio May 02 '22 at 10:25
5

This is what worked for me in AWS EKS cluster v1.21:

$ kubectl edit cm/kube-proxy-config -n kube-system
---
metricsBindAddress: 127.0.0.1:10249 ### <--- change to 0.0.0.0:10249
$ kubectl delete pod -l k8s-app=kube-proxy -n kube-system

Note, the name of the config map is kube-proxy-config, not kube-proxy

demisx
  • 7,217
  • 4
  • 45
  • 43
0

Both previous answers are correct, but if you are upgrading to eks 1.22, you could only upgrade the kube-proxy addon to v1.22.11-eksbuild.2 (current one) and the cm will be automatically updated

metricsBindAddress: 127.0.0.1:10249

to

metricsBindAddress: 0.0.0.0:10249

without the need to update it manually

You can see AWS kube-proxy addon documentation from https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html

Nidal
  • 1,717
  • 1
  • 27
  • 42