How to fix Cilium CrashLoopBackOff on minikube?

Question

I installed Meshery with Helm.

All pods are ok except the Cilium

meshery-app-mesh-7d8cb57c85-cfk7j                   1/1     Running            0                63m
meshery-cilium-69b4f7d6c5-t9w55                     0/1     CrashLoopBackOff   16 (3m16s ago)   63m
meshery-consul-67669c97cc-hxvlz                     1/1     Running            0                64m
meshery-istio-687c7ff5bc-bn8mq                      1/1     Running            0                63m
meshery-kuma-66f947b988-887hm                       1/1     Running            0                63m
meshery-linkerd-854975dcb9-rsgtl                    1/1     Running            0                63m
meshery-nginx-sm-55675f57b5-xfpgz                   1/1     Running            0                63m

Logs show

level=info msg="Registering workloads with Meshery Server for version v1.14.0-snapshot.4" app=cilium-adapter
panic: interface conversion: error is *fs.PathError, not *errors.Error

goroutine 42 [running]:
github.com/layer5io/meshkit/errors.GetCode({0x278e020?, 0xc0004f7e30?})

I checked events also

kubectl get events --field-selector involvedObject.name=meshery-cilium-69b4f7d6c5-t9w55
LAST SEEN   TYPE      REASON    OBJECT                                MESSAGE
17m         Normal    Pulling   pod/meshery-cilium-69b4f7d6c5-t9w55   Pulling image "layer5/meshery-cilium:stable-latest"
2m44s       Warning   BackOff   pod/meshery-cilium-69b4f7d6c5-t9w55   Back-off restarting failed container meshery-cilium in pod meshery-cilium-69b4f7d6c5-t9w55_default(d7ccd0e8-27e5-4f40-89bc-f8a6dc8fa25a)

I am adding logs from troubled pod Events:

  Type     Reason           Age                    From     Message
  ----     ------           ----                   ----     -------
  Normal   Pulling          4h25m (x55 over 8h)    kubelet  Pulling image "layer5/meshery-cilium:stable-latest"
  Warning  BackOff          3h45m (x1340 over 8h)  kubelet  Back-off restarting failed container meshery-cilium in pod meshery-cilium-69b4f7d6c5-t9w55_default(d7ccd0e8-27e5-4f40-89bc-f8a6dc8fa25a)
  Warning  FailedMount      21m (x6 over 21m)      kubelet  MountVolume.SetUp failed for volume "kube-api-access-tcdml" : object "default"/"kube-root-ca.crt" not registered
  Warning  NetworkNotReady  20m (x19 over 21m)     kubelet  network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  BackOff          72s (x82 over 19m)     kubelet  Back-off restarting failed container meshery-cilium in pod meshery-cilium-69b4f7d6c5-t9w55_default(d7ccd0e8-27e5-4f40-89bc-f8a6dc8fa25a)

The Nginx pod shows network problems

Events:
  Type     Reason           Age                 From             Message
  ----     ------           ----                ----             -------
  Warning  NodeNotReady     22m                 node-controller  Node is not ready
  Warning  NetworkNotReady  22m (x18 over 23m)  kubelet          network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
  Warning  FailedMount      22m (x7 over 23m)   kubelet          MountVolume.SetUp failed for volume "kube-api-access-lscv2" : object "default"/"kube-root-ca.crt" not registered

How to fix this issue?

score 1 · Answer 1 · answered Jul 13 '23 at 12:17

1

As per this Blog by Patrick Londa, your cilium Pod is getting CrashLoopBack off error due to the "Back-off restarting failed container", this means that your container suddenly terminated after Kubernetes started it.

1. This occurs due to the temporary resource overload as a result of a spike in activity. The solution to fix this is to adjust periodSeconds or timeoutSeconds to give the application a longer window of time to respond.

2. Sometimes this happens due to insufficient memory resources. You can increase the memory limit by changing the "resources:limits" in the Container's resource manifest as shown in the blog.

Verify the above steps and let me know if this fixes the issue.

answered Jul 13 '23 at 12:17

Hemanth Kumar

2,728
1
4
19

Take a look at my edit.pls! – Richard Rublev Jul 13 '23 at 12:52
1

As per this [SO](https://stackoverflow.com/a/67386884/19230181) by Sebastian Posada, he solved this issue by installing the [Pod network add-on](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network) this overlay network which satisfies the kubernetes requirements. Can you have a try and let me know if this fixes the issue. – Hemanth Kumar Jul 13 '23 at 13:09
Flannel does not change anything,it createsits own namespace. Meshery uses cilium. I will show all addons. – Richard Rublev Jul 13 '23 at 13:25
1

Refer to this [SO](https://stackoverflow.com/a/56344613/19230181) and [GIT Link](https://github.com/kubernetes/kubeadm/issues/1031) for more information. – Hemanth Kumar Jul 14 '23 at 04:49

How to fix Cilium CrashLoopBackOff on minikube?

1 Answers1