4

I experienced slow response of any kubectl command at my test environment. I don't have this issue in the production and at minikube in my dev. For example I run:

kubectl get deployments

And I have to wait even one minute. If I break with Ctrl + C, and start again sometimes it respond immediately. And when I repeat command is again delayed.

Do you have any tips where I should dig in ? Could it be memory issue ? Or rather network issue? How can I monitor them ? I'm in ssh session to one of master .

Bobby Van Kenobi
  • 41
  • 1
  • 2
  • Are your master nodes behind a load balancer? Have you checked the logs of the apiserver processes on the masters? Have you looked at the etcd logs for stalls? (that's where my money says is the problem, but the load balancer can do weird stuff, too) Does it only happen with Deployments, or does `kubectl get nodes` do the same thing? – mdaniel Jun 18 '20 at 15:37
  • @mdaniel It's example command . It happens for any kubectl command. I noticed as well that some kube-system pods are not stable as well . There are several pods with status 'Evicted'. I didn't do installation of Kubernetes but indeed I heard that our Kubernetes environment use Load balancer. I need to find out how to check logs . I'm software developer and devops stuff is new thing for me . – Bobby Van Kenobi Jun 18 '20 at 19:33
  • StackOverflow isn't a good place for those questions, as this is for programming problems -- you'll want [Server Fault](https://serverfault.com/questions/tagged/kubernetes) and even then, it's for asking _pointed_ questions, and not "I inherited a kubernetes cluster, what do?" Good luck! – mdaniel Jun 18 '20 at 20:23
  • are you using Flannel CNI? – Nick Jun 19 '20 at 07:40
  • whats your dev and prod configurations? – Vit Mar 20 '21 at 00:31
  • 1
    did usolve it ? – cryptoparty May 14 '21 at 06:30
  • Is your issue resolved?If yes, can you post the procedure you've followed as Solution and accept it? – Bakul Mitra Oct 20 '21 at 08:19
  • I would say, just restart API Server that can help sometime. https://stackoverflow.com/questions/51666507/how-can-kube-apiserver-be-restarted/67629356#67629356 – Amit Raj Nov 25 '21 at 00:50
  • Your issue has been resolved if not please update. – Abhijith Chitrapu Mar 02 '22 at 12:44

2 Answers2

2

If someone is looking at the same issue, you can try out -v=6 option to check API response timings.

kubectl get deployments -v=6
patilnitin
  • 1,151
  • 10
  • 14
2

I had also slow response of any kubectl command at one of my testing environment. I checked API response timings by using

kubectl get deployments -v=6

and encounter the following error

Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

This error is caused by the metrics server being down for some error, or in case it has been deleted. We can check the list of api services using kubectl get apiservice. If we want to get rid of this error we can just removed the apiservice using kubectl delete apiservice as follows:

$ kubectl delete apiservice v1beta1.metrics.k8s.io
apiservice.apiregistration.k8s.io "v1beta1.metrics.k8s.io" deleted

This blog supports the same idea.

V-Q-A NGUYEN
  • 1,497
  • 16
  • 19