my kubernetes cluster is down after reboot

Question

After every reboot my kubernetes cluster does not work fine and I get

The connection to the server 192.168.1.4:6443 was refused - did you specify the right host or port?

I have 4 ubuntu on baremetal one of them is master and 3 worker and I turned off swap and disabled it. and I read somewhere I should run this command two solve it

sudo -i
swapoff -a
exit
strace -eopenat kubectl version

and it is work. But why this was happening?

check if `kubelet` & `docker` are up and running on each node... — Ivan Aracki, Apr 07 '19 at 08:41
@yasinlachini is it still actual? did you manage to fix it? any errors in kubelet logs? — A_Suh, Apr 09 '19 at 10:09
I have not rebooted it after i reboot it and get error i post my log — yasin lachini, Apr 09 '19 at 10:13
Yasin jan, I have added a new answer which has happened to me recently. I hope it works. I would be thankful if you could accept my answer in case it is helpful. Thanks! — Mostafa Ghadimi, Aug 21 '22 at 14:40

score 7 · Answer 1 · edited Apr 08 '19 at 18:05

7

First please run systemctl status kubelet and verify if the service is running:
"Active: active (running)"
Disable swap:

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

verify all reference found in /etc/fstab about swap.

Please perform also post "kubeadm init" steps for current user as described here: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

After reboot please check:
systemctl status docker enable docker at startup if it's not working
systemctl enable docker

You can also verify kubelet status:

systemctl status kubelet
systemctl enable kubelet

take a look for any errors:

journalctl -u kubelet.service
journalctl

And please share with your findings.

edited Apr 08 '19 at 18:05

MWZ

1,224
7
11

answered Apr 08 '19 at 09:15

Mark

3,644
6
23

i write it blod that i turn it off and disable it – yasin lachini Apr 08 '19 at 10:37
of course i run those command if i do not run them the kubernetes does not work – yasin lachini Apr 08 '19 at 10:38
Hi Yasin, answer was updated about kubelet and docker status. – Mark Apr 08 '19 at 11:30

score 0 · Answer 2 · answered Apr 07 '19 at 06:53

0

Most likely that Kubelet is not getting restarted. You need to check Kubelet logs correct the issues if any.

Check docker driver and the driver used by kubelet should be same.

Swap should be disabled, and so on

answered Apr 07 '19 at 06:53

P Ekambaram

15,499
7
34
59

Mostafa Ghadimi · Answer 3 · 2022-08-21T21:59:58.030

It depends on how you install the cluster.

In this post, I will mention the possible ways to resolve this problem.

Make sure the swap is off.
```
swapoff -a
```
Check the state of Kubelet. In case it is exited and can't work properly, you can check the log of it.
```
journalctl -xfu kubelet.service
```
In my case the log wasn't that helpful. I was trying to see the log of other components. After searching and try to find a clue, I have found that there is an error with ‍cri-dockerd service, because it may not be enabled in systemd.
```
systemctl start cri-dockerd.service
systemctl enable cri-dockerd.service
```
Finally restart the kubelet service and check its status:
```
systemctl restart kubelet.service
systemctl status kubelet.service
```

my kubernetes cluster is down after reboot

3 Answers3