1

I am setting up the kubernetes cluster on CentOS 8 with containerd and Calico as CNI. with kubeadm command setup the master node, its in Ready status.

When I join the node to master, node not becoming ready status. I see below message the log file.

Jan 14 20:17:29 node02 containerd[1417]: time="2020-01-14T20:17:29.416373526-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-fbst8,Uid:9c7f6334-d106-48e1-af12-1bcdebc7c2c2,Namespace:kube-system,Attempt:0,} failed, error" error="failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown"
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416668   30113 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416742   30113 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416761   30113 kuberuntime_manager.go:729] createPodSandbox for pod "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416819   30113 pod_workers.go:191] Error syncing pod 9c7f6334-d106-48e1-af12-1bcdebc7c2c2 ("calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)"), skipping: failed to "CreatePodSandbox" for "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" with CreatePodSandboxError: "CreatePodSandbox for pod \"calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)\" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\\\"\": unknown"
Jan 14 20:17:30 node02 containerd[1417]: time="2020-01-14T20:17:30.541254039-05:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jan 14 20:17:30 node02 kubelet[30113]: E0114 20:17:30.541394   30113 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Jan 14 20:17:35 node02 containerd[1417]: time="2020-01-14T20:17:35.541792325-05:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jan 14 20:17:35 node02 kubelet[30113]: E0114 20:17:35.541929   30113 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Any tips to resolve this error?

sfgroups
  • 18,151
  • 28
  • 132
  • 204

2 Answers2

3

Did you setting --pod-network-cidr=192.168.0.0/16 to kubeadm init?
Apparently, You need setting it.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

bells17
  • 370
  • 2
  • 9
  • yes, I have this in my kubeadm configuration file. I have setup few cluster with docker. this is the first time using `containerd` – sfgroups Jan 15 '20 at 02:13
  • I tried & succeeded by [these commands](https://gist.github.com/bells17/3f00e5ca42e54845da0076b35ff9235f). – bells17 Jan 15 '20 at 02:44
  • I missed setting cgroup args for kubelet, after setting it. I was able to join the node. Thanks for the help. – sfgroups Jan 15 '20 at 13:03
2

Because you are not using docker you need to setup the cgroup driver explicitly.

To use the systemd cgroup driver, set plugins.cri.systemd_cgroup = true in /etc/containerd/config.toml and systemctl restart containerd

You have to modify the file kubeadm-flags.env in /var/lib/kubelet and set the cgroups driver.

KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

Make sure to point to above file in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
jmcgrath207
  • 1,317
  • 2
  • 19
  • 31
Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107