Getting "CrashLoopBackOff" as status of deployed pod

Question

How to debug why it's status is CrashLoopBackOff?

I am not using minikube , working on Aws Kubernetes instance.

I followed this tutorial. https://github.com/mkjelland/spring-boot-postgres-on-k8s-sample

When I do

  kubectl create -f specs/spring-boot-app.yml

and check status by

  kubectl get pods

it gives

     spring-boot-postgres-sample-67f9cbc8c-qnkzg   0/1     CrashLoopBackOff   14         50m

Below Command

 kubectl describe pods spring-boot-postgres-sample-67f9cbc8c-qnkzg

gives

Events:
  Type     Reason   Age                    From                      Message
  ----     ------   ----                   ----                      -------
  Warning  BackOff  3m18s (x350 over 78m)  kubelet, ip-172-31-11-87  Back-off restarting failed container

Command kubectl get pods --all-namespaces gives

NAMESPACE     NAME                                           READY   STATUS             RESTARTS   AGE
default       constraintpod                                  1/1     Running            1          88d
default       postgres-78f78bfbfc-72bgf                      1/1     Running            0          109m
default       rcsise-krbxg                                   1/1     Running            1          87d
default       spring-boot-postgres-sample-667f87cf4c-858rx   0/1     CrashLoopBackOff   4          110s
default       twocontainers                                  2/2     Running            479        89d
kube-system   coredns-86c58d9df4-kr4zj                       1/1     Running            1          89d
kube-system   coredns-86c58d9df4-qqq2p                       1/1     Running            1          89d
kube-system   etcd-ip-172-31-6-149                           1/1     Running            8          89d
kube-system   kube-apiserver-ip-172-31-6-149                 1/1     Running            1          89d
kube-system   kube-controller-manager-ip-172-31-6-149        1/1     Running            1          89d
kube-system   kube-flannel-ds-amd64-4h4x7                    1/1     Running            1          89d
kube-system   kube-flannel-ds-amd64-fcvf2                    1/1     Running            1          89d
kube-system   kube-proxy-5sgjb                               1/1     Running            1          89d
kube-system   kube-proxy-hd7tr                               1/1     Running            1          89d
kube-system   kube-scheduler-ip-172-31-6-149                 1/1     Running            1          89d

Command kubectl logs spring-boot-postgres-sample-667f87cf4c-858rx doesn't print anything.

you can check pods using `kubectl describe pods spring-boot-postgres-sample-67f9cbc8c-qnkzg` — Thanh Nguyen Van, May 23 '19 at 08:59
@ThanhNguyenVan Edited ..i added Events :., anything else you want me to add? — Dhanraj, May 23 '19 at 09:07
`docker logs container_id` to show logs of container belong to that pods — Thanh Nguyen Van, May 23 '19 at 09:30
Sorry i don't know how to get container_id...I tried **kubectl logs /jarName** but no such container — Dhanraj, May 23 '19 at 09:38
Run `kubectl describe pods spring-boot-postgres-sample-67f9cbc8c-qnkzg` it show container belong to. — Thanh Nguyen Van, May 23 '19 at 09:39
It gives under container section : - **Container ID: docker://c9704202d90fbfa87274d59dd571d2084de2b969db2be20aaffe5f0801b5f52d** — Dhanraj, May 23 '19 at 09:47
`docker logs c9704202d90fbfa87274d59dd571d2084de2b969db2be20aaffe5f0801b5f52d` — Thanh Nguyen Van, May 23 '19 at 09:47
I did **sudo docker logs c9704202d90fbfa87274d59dd571d2084de2b969db2be20aaffe5f0801b5f52d** but it gives no such container_id — Dhanraj, May 23 '19 at 09:48
run `kubectl describe pods` again. get new container_id then show its logs — Thanh Nguyen Van, May 23 '19 at 09:51
Please run same docker image with docker ex. docker run and see — Yash Jagdale, May 23 '19 at 10:15
are you able to run the docker image on your local machine? maybe you forgot to change the image name in the yml file? — Amityo, May 23 '19 at 10:58
can you run this command and show the output: kubectl get pods --all-namespaces and kubectl logs spring-boot-postgres-sample-67f9cbc8c-qnkzg to get the logs of pod — Anshul Jindal, May 23 '19 at 11:04
Have you tried running this application locally without Kubernetes ? — Anshul Jindal, May 23 '19 at 14:18
I build it on my window machine,then put that jar in kubernetes instance..and from there to created image to docker — Dhanraj, May 23 '19 at 14:21

Dhanraj · Accepted Answer · 2019-05-24T07:36:25.910

First of all I fixed by postgres deployment, there was some error of "pod has unbound PersistentVolumeClaims" , so i fixed that error by this post pod has unbound PersistentVolumeClaims

So now my postgres deployment is running.

kubectl logs spring-boot-postgres-sample-67f9cbc8c-qnkzg doesn't print anything, it means there is something wrong in config file. kubectl describe pod spring-boot-postgres-sample-67f9cbc8c-qnkzg stating that container is terminated and reason is completed, I fixed it by running container infinity time by adding

   # Just sleep forever
command: [ "sleep" ]
args: [ "infinity" ]

So now my deployment is running. But now i Exposed my service by

kubectl expose deployment spring-boot-postgres-sample --type=LoadBalancer --port=8080

but can't able to get External-Ip , so I did

kubectl patch svc <svc-name> -n <namespace> -p '{"spec": {"type": "LoadBalancer", "externalIPs":["172.31.71.218"]}}'

So I get my external-Ip as "172.31.71.218"

But now the problem is curl http://172.31.71.218:8080/ getting timeout

Anything i did wrong?

Here is my deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: spring-boot-postgres-sample
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      name: spring-boot-postgres-sample
      labels:
        app: spring-boot-postgres-sample
    spec:
      containers:
      - name: spring-boot-postgres-sample
        command: [ "/bin/bash", "-ce", "tail -f /dev/null" ]
        env:
          - name: POSTGRES_USER
            valueFrom:
              configMapKeyRef:
                name: postgres-config
                key: postgres_user
          - name: POSTGRES_PASSWORD
            valueFrom:
              configMapKeyRef:
                name: postgres-config
                key: postgres_password
          - name: POSTGRES_HOST
            valueFrom:
              configMapKeyRef:
                name: hostname-config
                key: postgres_host
        image: <mydockerHUbaccount>/spring-boot-postgres-on-k8s:v1

Here is my postgres.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  namespace: default
data:
  postgres_user: postgresuser
  postgres_password: password
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: postgres
spec:
  template:
    metadata:
      labels:
        app: postgres
    spec:
      volumes:
        - name: postgres-storage
          persistentVolumeClaim:
            claimName: postgres-pv-claim
      containers:
        - image: postgres
          name: postgres
          env:
            - name: POSTGRES_USER
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: postgres_user
            - name: POSTGRES_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: postgres-config
                  key: postgres_password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - containerPort: 5432
              name: postgres
          volumeMounts:
            - name: postgres-storage
              mountPath: /var/lib/postgresql/data
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  type: ClusterIP
  ports:
    - port: 5432
  selector:
    app: postgres

Here How i got host-config map

kubectl create configmap hostname-config --from-literal=postgres_host=$(kubectl get svc postgres -o jsonpath="{.spec.clusterIP}")

Can't able to get logs..does it mean still my config is wrong?can you please check my deployment.yaml.. @ThanhNguyenVan — Dhanraj, May 24 '19 at 07:20
Yes . I did ... I also added my postgres.config and host-config — Dhanraj, May 24 '19 at 07:36

score 2 · Answer 2 · answered May 23 '19 at 15:21

2

Why don't you...

run a dummy container (run an endless sleep command)
kubectl exec -it bash
Run the program directly and have a look at the logs directly.

Its an easier form of debugging on K8s.

answered May 23 '19 at 15:21

DUDANF

2,618
1
12
42

I have spring boot jar, what do you mean by run program directly..and how to look at their logs? – Dhanraj May 23 '19 at 15:25
Does this not show you the output? ```bash kubectl logs ``` – DUDANF May 24 '19 at 09:09

score 0 · Answer 3 · answered May 23 '19 at 11:03

I was able to reproduce the scenario. Seems there is a connectivity issue between the app and Postgres DB. So the app failed to initiate. Please find the logs below it might help you.

$ kubectl get po
NAME                                           READY     STATUS             RESTARTS   AGE
spring-boot-postgres-sample-5d7c85d98b-qwvjr   0/1       CrashLoopBackOff   19         1h


org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaAutoConfiguration.class]: Invocation of init method failed; nested exception is org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]

2019-05-23 10:53:01.889 ERROR 1 --- [           main] o.a.tomcat.jdbc.pool.ConnectionPool      : Unable to create initial connections of pool.

org.postgresql.util.PSQLException: Connection to :5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51) ~[postgresql-9.4.1212.jre7.jar!/:9.4.1212.jre7]

`kubectl logs spring-boot-postgres-sample-5d7c85d98b-hl5z2 -f` — hariK, May 23 '19 at 12:17

Getting "CrashLoopBackOff" as status of deployed pod

3 Answers3