Error from server (NotFound): podmetrics.metrics.k8s.io "mem-example/memory-demo" not found

Question

I am following this tutorial: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/

I have created the memory pod demo and I am trying to get the metrics from the pod but it is not working.

I installed the metrics server by cloning: https://github.com/kubernetes-incubator/metrics-server

And then running this command from top level:

kubectl create -f deploy/1.8+/

I am using kubernetes version 1.10.11.

The pod is definitely created:

λ kubectl get pod memory-demo --namespace=mem-example
NAME          READY     STATUS    RESTARTS   AGE
memory-demo   1/1       Running   0          6m

But the metics command does not work and gives an error:

λ kubectl top pod memory-demo --namespace=mem-example
Error from server (NotFound): podmetrics.metrics.k8s.io "mem-example/memory-demo" not found

What did I do wrong?

score 7 · Answer 1 · edited Jun 20 '20 at 09:12

There are some patches to be done to metrics server deployment to get the metrics working.

Follow the below steps

kubectl delete -f deploy/1.8+/

wait till the metrics server gets undeployed

run the below command
kubectl create -f https://raw.githubusercontent.com/epasham/docker-repo/master/k8s/metrics-server.yaml

master $ kubectl get po -n kube-system
NAME                              READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-6zg78          1/1       Running   0          2h
coredns-78fcdf6894-gk4sb          1/1       Running   0          2h
etcd-master                       1/1       Running   0          2h
kube-apiserver-master             1/1       Running   0          2h
kube-controller-manager-master    1/1       Running   0          2h
kube-proxy-f5z9p                  1/1       Running   0          2h
kube-proxy-ghbvn                  1/1       Running   0          2h
kube-scheduler-master             1/1       Running   0          2h
metrics-server-85c54d44c8-rmvxh   2/2       Running   0          1m
weave-net-4j7cl                   2/2       Running   1          2h
weave-net-82fzn                   2/2       Running   1          2h

master $ kubectl top pod -n kube-system
NAME                              CPU(cores)   MEMORY(bytes)
coredns-78fcdf6894-6zg78          2m           11Mi
coredns-78fcdf6894-gk4sb          2m           9Mi
etcd-master                       14m          90Mi
kube-apiserver-master             24m          425Mi
kube-controller-manager-master    26m          62Mi
kube-proxy-f5z9p                  2m           19Mi
kube-proxy-ghbvn                  3m           17Mi
kube-scheduler-master             8m           14Mi
metrics-server-85c54d44c8-rmvxh   1m           19Mi
weave-net-4j7cl                   2m           59Mi
weave-net-82fzn                   1m           60Mi

Check and verify the below lines in metrics server deployment manifest.

        command:
        - /metrics-server
        - --metric-resolution=30s
        - --kubelet-preferred-address-types=InternalIP
        - --kubelet-insecure-tls

That doesn't look like an official repo. I am wary of installing random code. Do you have a reference to the patches you are talking about? — Guerrilla, Feb 25 '19 at 12:55
It was taken from official report and patches were added on top of that — P Ekambaram, Feb 25 '19 at 13:15
updated the changes that you need to apply to metrics server. check my answer above — P Ekambaram, Feb 25 '19 at 13:27

score 6 · Answer 2 · answered Aug 12 '20 at 00:53

6

On Minikube, I had to wait for 20-25 minutes after enabling the metrics-server addon. I was getting the same error for 20-25 minutes but later I could see the output without attempting for any solution.

answered Aug 12 '20 at 00:53

Dean Winchester

352
3
7

1

Me too. It's even in the documentation: `kubectl top pod --help` says: Due to the metrics pipeline delay, they may be unavailable for a few minutes since pod creation. – bronson May 06 '21 at 22:03

score 1 · Answer 3 · edited May 19 '20 at 18:52

I faced the similar issue of

Error from server (NotFound): podmetrics.metrics.k8s.io "default/apple-app" not found

I followed two steps and I was able to resolve the issue.

Download the latest customized components.yaml, which is their official file used for easy deployment.
Update the change

          # - /metrics-server
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP

to the command section of the deployment specification. I have commented the first line because it is the entrypoint of the image used by kubernetes metrics-server.

$ docker image inspect k8s.gcr.io/metrics-server-amd64:v0.3.6 -f {{.ContainerConfig.Entrypoint}}
[/metrics-server]

Even If you use it or not, it doesn't matter.

Note: You have to wait for few seconds for it to properly work.

After this running the top command will work for you.

$ kubectl top pod apple-app
NAME        CPU(cores)   MEMORY(bytes)   
apple-app   1m           3Mi

score 1 · Answer 4 · edited Jul 01 '20 at 00:43

I know this is an old thread may be someone will find this answer useful.

You have to checkout the following repo:

https://github.com/kubernetes-incubator/metrics-server

Go to the root of the repo and checkout release-0.3.2.
Remove default metrics server by:
```
kubectl delete -f deploy/1.8+/
```

Download the container yaml

wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

Edit the container.yaml by adding the following lines to the argument section. You will see these two lines there
```
args:
      - --kubelet-preferred-address-types=InternalIP
      - --kubelet-insecure-tls=true
```
There is only one args parameter in that file.

Deploy your pod/deployment and you should be able to do:

kubectl top pod <pod-name>

Error from server (NotFound): podmetrics.metrics.k8s.io "mem-example/memory-demo" not found

4 Answers4

Follow the below steps

Linked