Kubernetes containers CrashLoopBackOff

Question

I am new to Kubernetes and trying to learn but I am stuck with an error that I cannot find an explanation for. I am running Pods and Deployments in my cluster and they are running perfectly as shown in the CLI, but after a while they keep crashing and the Pods need to restart.

I did some research to fix my issue before posting here, but the way I understood it, I will have to make a deployment so that my replicaSets will manage my Pods lifecycle and not deploy Pods independently. But as you can see also Pods in deployment is crashing.

kubectl get pods

operator-5bf8c8484c-fcmnp   0/1     CrashLoopBackOff   9          34m
operator-5bf8c8484c-phptp   0/1     CrashLoopBackOff   9          34m
operator-5bf8c8484c-wh7hm   0/1     CrashLoopBackOff   9          34m
operator-pod                0/1     CrashLoopBackOff   12         49m

kubectl describe pods operator

Events:
  Type     Reason     Age                   From                 Message
  ----     ------     ----                  ----                 -------
  Normal   Scheduled  <unknown>             default-scheduler    Successfully assigned default/operator-pod to workernode
  Normal   Created    30m (x5 over 34m)     kubelet, workernode  Created container operator-pod
  Normal   Started    30m (x5 over 34m)     kubelet, workernode  Started container operator-pod
  Normal   Pulled     29m (x6 over 34m)     kubelet, workernode  Container image "operator-api_1:java" already present on machine
  Warning  BackOff    4m5s (x101 over 33m)  kubelet, workernode  Back-off restarting failed container

deployment yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
   name: operator
   labels:
     app: java
spec:
  replicas: 3
  selector:
    matchLabels:
      app: call
  template:
    metadata:
      labels:
        app: call
    spec:
       containers:
       - name: operatorapi
         image: operator-api_1:java
         ports:
         - containerPort: 80

Can someone help me out, how can I debug?

Do your pod have a command to run when they start ? I do not see a command specified on your deployment. If your Pod doesn't have a command to run , it will exit as soon as it start , that could explain the crashloopbackoff — Popopame, Jul 06 '20 at 14:33
@Popopame If one is not specified, it will default to ENTRYPOINT/COMMAND from the Dockerfile. — Amir, Jul 06 '20 at 20:21

score 1 · Answer 1 · answered Jul 06 '20 at 14:34

The reason is most probably the process running in container finished its task and terminated by container OS after a while. Then the pod is being restarted by kubelet.

What I recommend you to solve this issue, please check the process running in container and try to keep it alive forever. You can create a loop to run this process in container or you can use some commands for container on the deployment.yaml

Here is a reference for you to understand and debug pod failure reason. https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/

Kindly note that I checked the reference to debug about my Pods crashing, but there was nothing related to my issue as I do not have a termination message in my output. I will try to create a loop in my container as you mentioned. Thank you — amin224, Jul 07 '20 at 07:55

score 0 · Answer 2 · answered Jul 06 '20 at 14:47

There are several ways to debug such a scenario and I recommend viewing Kubernetes documentation for best-practices. I typically have success with the following 2 approaches:

Logs: You can view the logs for the application using the command below:

kubectl logs -l app=java

If you have multiple containers within that pod, you can filter it down:

kubectl logs -l app=java -c operatorapi

Events: You can get a lot of information from events as shown below (sorted by timestamp). Keep in mind that there could be a lot of noise in events depending on the number of apps and services that you may have so you have to filter it down further:

kubectl get events --sort-by='.metadata.creationTimestamp'

Feel free to share the output from those two and I can help you debug further.

Thank you Amir for your help. I added 3 more containers from a simple application that should not crash, unfortunately the issue still remained with the same events message similar to the above post (Back-off restarting failed container). Do I need to create a loop in the deployment file so that my Pods do not crash? — amin224, Jul 07 '20 at 09:45

Kubernetes containers CrashLoopBackOff

2 Answers2