Metrics server not working in Kubernetes cluster

Question

I have set up kubernetes cluster on ubuntu 18+. It's working fine. now i had added metric server but it's not working.

# kubectl get apiservices

v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (FailedDiscoveryCheck)   2d1h

    # kubectl describe apiservice v1beta1.metrics.k8s.io


    Message:               failing or missing response from https://10.106.145.77:443/apis/metrics.k8s.io/v1beta1: Get https://10.106.145.77:443/apis/metrics.k8s.io/v1beta1: dial tcp 10.106.145.77:443: connect: connection refused
    Reason:                FailedDiscoveryCheck

I don't know why connection is refused. can any one help me or give me some hint to fix this. i have added RBAC, in cluster, does that is issue? I have try many solution from net but no one can help me. I had try to edit deployment yaml of metric server with args and insecure TLS but didn't help.

Other Details

# kubectl get all --all-namespaces | grep -i metrics-server

kube-system            pod/metrics-server-7f55d7ccbb-th9w9              1/1     Running   0          21s
kube-system            service/metrics-server              ClusterIP   10.106.145.77    <none>        443/TCP                                         26m
kube-system            deployment.apps/metrics-server              1/1     1            1           25m
kube-system            replicaset.apps/metrics-server-694db48df9              0         0         0       25m
kube-system            replicaset.apps/metrics-server-7f55d7ccbb              1         1         1       21s


# kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10

 - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: Always
        name: metrics-server
        ports:
        - containerPort: 4443
          hostPort: 4443

Yml File :-

# kubectl get -n kube-system deployment metr                                                                                                                     ics-server -o yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "2"
      creationTimestamp: "2020-01-29T14:49:06Z"
      generation: 2
      labels:
        k8s-app: metrics-server
      name: metrics-server
      namespace: kube-system
      resourceVersion: "951901"
      selfLink: /apis/apps/v1/namespaces/kube-system/deployments/metrics-server
      uid: 54137f75-af0a-45a5-a508-f4c38ee9ea25
    spec:
      progressDeadlineSeconds: 600
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          k8s-app: metrics-server
      strategy:
        rollingUpdate:
          maxSurge: 25%
          maxUnavailable: 25%
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            k8s-app: metrics-server
          name: metrics-server
        spec:
          containers:
          - args:
            - --cert-dir=/tmp
            - --secure-port=4443
            - --kubelet-insecure-tls
            - --kubelet-preferred-address-types=InternalIP
            image: k8s.gcr.io/metrics-server-amd64:v0.3.6
            imagePullPolicy: Always
            name: metrics-server
            ports:
            - containerPort: 4443
              hostPort: 4443
              name: main-port
              protocol: TCP
            resources: {}
            securityContext:
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              runAsUser: 1000
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /tmp
              name: tmp-dir
          dnsPolicy: ClusterFirst
          hostNetwork: true
          nodeSelector:
            beta.kubernetes.io/os: linux
            kubernetes.io/arch: amd64
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          serviceAccount: metrics-server
          serviceAccountName: metrics-server
          terminationGracePeriodSeconds: 30
          volumes:
          - emptyDir: {}
            name: tmp-dir
    status:
      availableReplicas: 1
      conditions:
      - lastTransitionTime: "2020-01-29T14:49:15Z"
        lastUpdateTime: "2020-01-29T14:49:15Z"
        message: Deployment has minimum availability.
        reason: MinimumReplicasAvailable
        status: "True"
        type: Available
      - lastTransitionTime: "2020-01-29T14:49:06Z"
        lastUpdateTime: "2020-01-29T15:14:26Z"
        message: ReplicaSet "metrics-server-7f55d7ccbb" has successfully progressed.
        reason: NewReplicaSetAvailable
        status: "True"
        type: Progressing
      observedGeneration: 2
      readyReplicas: 1
      replicas: 1
      updatedReplicas: 1

what do you see when you run for say this command `$ kubectl get all --all-namespaces | grep -i metrics-server` .. is the deployment/pod/service listed correctly with no issues !! — DT., Jan 29 '20 at 15:01
have you also added `--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname` to the metric server on deployment time !! if you could share the output of `$ kubectl get -n kube-system deployment metrics-server -o yaml | grep -i args -A 10` on what arguments you have used. — DT., Jan 29 '20 at 15:08
@D.T. Please find required details, i have put output of them — Vinit Patel, Jan 29 '20 at 15:18
Please confirm the metric-server version used is it 1.8+ or lower.. Refer This [metrics-server is not accessible when rbac is enabled](https://github.com/ubuntu/microk8s/issues/560) — DT., Jan 29 '20 at 16:02
Could you describe your environment? Is it hosted in cloud? Are you behind proxy or anything that might block connection? — kool, Jan 29 '20 at 17:29
@kfc it's in local network machine. Not on cloud and no proxy — Vinit Patel, Jan 29 '20 at 18:04
Can you add `- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname` to your metrics-server deployment? — kool, Jan 29 '20 at 18:39

score 5 · Accepted Answer · edited Dec 03 '21 at 08:52

5

Find the args section and try this. Adding the command and /metrics server solved my problem along with updating the preferred address type and then restarting kubelet.

args:
        - --cert-dir=/tmp
        - --secure-port=4443
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname

edited Dec 03 '21 at 08:52

Marc

13,011
11
78
98

answered Sep 23 '20 at 13:39

RecentCoin

316
3
5

1

SO doesn't allow me to edit your post so I drop it as comment: There is a typo in `External IP` it *should* be `ExternalIP`and if you mind improving the indentation command keyword, would also help making it more readable – JonasP Nov 04 '20 at 11:02
2

Instead of using `--kubelet-insecure-tls` you should ensure that the kubelet server receives certificates signed by the kubernetes CA. Have a look at those issues: https://github.com/kubernetes/kubeadm/issues/1223#issuecomment-619043303 and https://github.com/kubernetes-sigs/metrics-server/issues/146#issuecomment-459239615 – dahrens Dec 30 '20 at 18:52

score 1 · Answer 2 · edited Feb 19 '21 at 01:22

Faced similar issue with 503 Service Unavailable error message. Managed to fix the issue by making below changes.

On your components.yaml file, make sure cert path is correct:

-- cert-dir=/etc/kubernetes/pki

kubectl apply -f components.yaml

(change cert path to that instead of default /tmp. It may depend on your setup so, try find out where your pki certs are on your machine. Mine was on /etc/kubernetes/pki)

Metrics server not working in Kubernetes cluster

2 Answers2