52

I am trying to run Kubernetes and trying to use sudo kubeadm init. Swap is off as recommended by official doc.

The issue is it displays the warning:

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.


Unfortunately, an error has occurred:
            timed out waiting for the condition

This error is likely caused by:
            - The kubelet is not running
            - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
            - No internet connection is available so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-amd64:v1.11.2
                - k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
                - k8s.gcr.io/kube-scheduler-amd64:v1.11.2
                - k8s.gcr.io/etcd-amd64:3.2.18
                - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
                  are downloaded locally and cached.

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
            - 'systemctl status kubelet'
            - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
        Here is one example how you may list all Kubernetes containers running in docker:
            - 'docker ps -a | grep kube | grep -v pause'
            Once you have found the failing container, you can inspect its logs with:
            - 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

The docker version I am using is Docker version 17.03.2-ce, build f5ec1e2 I m using Ubuntu 16.04 LTS 64bit

The docker images shows the following images:

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver-amd64            v1.11.2             821507941e9c        3 weeks ago         187 MB
k8s.gcr.io/kube-controller-manager-amd64   v1.11.2             38521457c799        3 weeks ago         155 MB
k8s.gcr.io/kube-proxy-amd64                v1.11.2             46a3cd725628        3 weeks ago         97.8 MB
k8s.gcr.io/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        3 weeks ago         56.8 MB
k8s.gcr.io/coredns                         1.1.3               b3b94275d97c        3 months ago        45.6 MB
k8s.gcr.io/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219 MB
k8s.gcr.io/pause                           3.1                 da86e6ba6ca1        8 months ago        742 kB

Full logs can be found here : https://pastebin.com/T5V0taE3

I didn't found any solution on internet.

EDIT:

docker ps -a output:

ubuntu@ubuntu-HP-Pavilion-15-Notebook-PC:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS

journalctl -xeu kubelet output:

journalctl -xeu kubelet
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: T
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: M
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: U
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: F
lines 788-810/810 (END)
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117131    9107 server.go:408] Version: v1.11.2
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117406    9107 plugins.go:97] No cloud provider specified.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.121192    9107 certificate_store.go:131] Loading cert/key pair 
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.145720    9107 server.go:648] --cgroups-per-qos enabled, but --
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.146074    9107 server.go:262] failed to run Kubelet: Running wi
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Unit entered failed state.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Failed with result 'exit-code'.
~


          PORTS               NAMES

Any help/suggestion/comment would be appreciated.

Akshay Sood
  • 6,366
  • 10
  • 36
  • 59

12 Answers12

103

I faced similar issue recently. The problem was cgroup driver. Kubernetes cgroup driver was set to systems but docker was set to systemd. So I created /etc/docker/daemon.json and added below:

{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

Then

 sudo systemctl daemon-reload
 sudo systemctl restart docker
 sudo systemctl restart kubelet

Run kubeadm init or kubeadm join again.

Shakiba Moshiri
  • 21,040
  • 2
  • 34
  • 44
Ishan Mahadevia
  • 1,031
  • 2
  • 4
  • 2
  • 3
    It's working for me in AWS Ubuntu instance. – Miguel Conde Aug 18 '21 at 16:30
  • that's cool.. wow it's working in Digital Ocean VM instance too thanks bro – anztrax Aug 19 '21 at 08:49
  • 1
    it works in a baremetal equinix server too! ty man! – Math Sep 11 '21 at 15:07
  • It works for me on AWS RHEL 7 , Thanks alot! – Ritesh Singh Oct 12 '21 at 16:49
  • It works for me on ubuntu 18.04, awsome~~ – Scorpioooooon21 Dec 07 '21 at 13:04
  • 2
    Wow!!!! Worked for me on Ubuntu 20.04 – Arash Foroughi Jan 06 '22 at 21:25
  • 1
    Awesome! Fixed the problem for me on Ubuntu 20.04 on a Hyper-V VM. Saved me a LOT of time and frustration! The only thing I found was I had to revert to a checkpoint taken before running kubeadm the first time. Re-running kubeadm init after making these changes failed preflight checks due to ports in use etc etc. I perhaps could have fixed those, but I had the checkpoint, so... :) – Deltics Jan 11 '22 at 07:39
  • 2
    You can diagnose the problem by running the suggested command; `journalctl -xeu kubelet`. In this command you will something like the following log; `err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""` – zaf187 Jan 25 '22 at 11:14
  • Works like magic – Stan Apr 03 '22 at 09:55
44

Unfortunately, swap was still enabled.
The error was fixed by

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

reboot the machine.

Akshay Sood
  • 6,366
  • 10
  • 36
  • 59
18

I had this same issue when trying to initialize my k8s cluster. In my case, the error was rooting from Docker and Kubelet having inconsistent cgroups.

To solve it, first find Docker cgroup:

docker info | grep Cgroup

The result of the above command would be something like this:

Cgroup Driver: cgroupfs
Cgroup Version: 1

Then, update kubelet args (KUBELET_KUBECONFIG_ARGS) in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and add a --cgroup-driver flag corresponsing to docker cgroup (in this case cgroupfs).

My config file looks like this after the modification:

...
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/e`tc/kubernetes/kubelet.conf --cgroup-driver=cgroupfs"
...

Finally, run kubeadm reset and then kubeadm init.

Hamid Mohayeji
  • 3,977
  • 3
  • 43
  • 55
5

I met the exactly same problem.Turn swap off can solve this problem. But I avoided it in a different way, so I post my solution here for reference.

Swap check in kubelet have two phases. One is kubeadm command line tool,the other one is kubelet service.

So if we don't turn swap off,kubeadm should show the message below

[ERROR Swap]: running with swap on is not supported. Please disable swap

then it will abort the init or join process. By adding the parameter "--ignore-preflight-errors=Swap" to kubeadm can avoid kubeadm check. Here is an example:

sudo kubeadm join 10.50.10.198:6443 --token XXXX.XXXXXa     --discovery-token-ca-cert-hash sha256:XX48cb7c381 --ignore-preflight-errors=Swap

However, if we run this. Kubelet service will block us ,and it will occur the problem of this thread. We can add a configuration file to avoid this:

 cd /etc/systemd/system/kubelet.service.d
 touch 20-allow-swap.conf

Add these contents into this file.

[Service] 
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false”

Then

 systemctl daemon-reload
 systemctl restart kubelet

At last don't forget to run kubeadm join agin,with "--ignore-preflight-errors=Swap"

bluezealot
  • 99
  • 1
  • 4
4

sudo sed -i '/ swap / s/^/#/' /etc/fstab

I spent last few days to solve the same problem and this also worked for me. Don't know whay it doesn't have same effect only with 'sudo swapoff -a'.

[kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

And these error logs are really confusing because there is no clue there showing that this problem may be related to swap configuraiton.

late spring
  • 49
  • 1
  • 2
2

As you use Kubernetes 1.11.2, it's useful to quote from CHANGELOG-1.11.md:

kubeadm now detects the Docker cgroup driver and starts the kubelet with the matching driver. This eliminates a common error experienced by new users in when the Docker cgroup driver is not the same as the one set for the kubelet due to different Linux distributions setting different cgroup drivers for Docker, making it hard to start the kubelet properly.

It seems to me that on your node some wrong parameter is passed to kubelet and this one doesn't start.

  1. First of all check that your docker has cfgroups with the command:

    docker info | grep -i cgroup

Output should be:

Cgroup Driver: cgroupfs

  1. Now go on your node and look for the kubelet service script, probably in /etc/systemd/system/kubelet.service (or similar name) and remove all parameters regarding cfgroup inside that script.

  2. Try to restart the kubelet service

  3. Check the kubelet's log again (journalctl -xeu kubelet)

tgogos
  • 23,218
  • 20
  • 96
  • 128
Nicola Ben
  • 10,615
  • 8
  • 41
  • 65
  • 2
    There is a file named `10-kubeadm.conf` in `/etc/systemd/system/kubelet.service.d`. But there is no parameter regarding `cfgroup` inside – Akshay Sood Sep 02 '18 at 09:05
  • can you post the content? Your previous `journalctl -xeu kubelet` output was cut. If you can reedit, it would be clearer. – Nicola Ben Sep 02 '18 at 09:06
  • There are references to these ones: config.yaml , kubeadm-flags.env, /etc/default/kubelet, take a look there. – Nicola Ben Sep 02 '18 at 09:12
  • one more thing I want to tell you. If I run same command on Oracle Virtual Machine Ubuntu 16.04 64Bit minimal image (54 mb) , `sudo kubeadm init` works fine – Akshay Sood Sep 02 '18 at 09:12
  • Docker installation were done by you on both machines? – Nicola Ben Sep 02 '18 at 09:14
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/179266/discussion-between-nicola-ben-and-akshay-sood). – Nicola Ben Sep 02 '18 at 09:14
2

use

export KUBECONFIG=/etc/kubernetes/admin.conf

or you can create file name

nano  /etc/docker/daemon.json

and write

{
      "exec-opts": ["native.cgroupdriver=cgroupfs"],
      "log-driver": "json-file",
      "log-opts": {
      "max-size": "100m"
   },
       "storage-driver": "overlay2"
       }

then

systemctl daemon-reload
systemctl restart kubelet
systemctl status  kubelet
Shihab Uddin Shakil
  • 85
  • 3
0

if you are running on openstack or any cloud please make sure to allow the ports in security groups TCP | Inbound | 6443* | Kubernetes API server

in my case this was the error

jobin at
  • 1
0

You should not use with swap even have been closed by swapoff -a.

You should disable in setting file /etc/fstab

for me:

root@kali:~# cat /etc/fstab 


UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

after editing:

root@kali:~# cat /etc/fstab 

UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

#UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

then.. reboot your machine

Note: Don't use reset after rebooting before kubeadm init via:

kubeadm reset

dose
  • 43
  • 1
  • 3
0

You can try following:

sudo apt-get install -yq kubelet=1.25.5-00 kubeadm=1.25.5-00
Tyler2P
  • 2,324
  • 26
  • 22
  • 31
sglai
  • 307
  • 4
  • 10
  • I flagged this for conversion to a comment or [deletion](/help/deleted-answers) because it is not an answer. Once you [earn](//meta.stackoverflow.com/q/146472) enough [reputation](/help/whats-reputation), you will gain the privilege to [add comments](/help/privileges/comment) in addition to answers. Then you will be able to request clarification, add related links, leave constructive criticism, and address other users directly. – Stephen Ostermiller Dec 30 '22 at 11:12
  • for me no other solution worked other than this, downgrading packages – richa verma Jan 14 '23 at 17:49
-1

I had to modify ExecStart options in /usr/lib/systemd/system/docker.service as indicated in one of the responses here: docker change cgroup driver to systemd

And very important: I had to delete the file /etc/docker/daemon.json that I had created trying to solve this problem. Otherwise, after modifying /usr/lib/systemd/system/docker.service you get an error because the same option is also in daemon.json

rodolk
  • 5,606
  • 3
  • 28
  • 34
-1

create /etc/docker/daemon.json

and add code below and restart docker

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
khoa junior
  • 69
  • 3