4

Trying to solve dependency between pods using postStart lifecycle.

Use case: micro service A should start after the start of micro service B.

For that we have added one container (curl) which will check if dependent service is up or not using curl command.

But when we add any command in postStart lifecycle hook, pods keep restarting and goes in crashlookbackoff state

Deployment.yaml :

kind: Deployment
metadata:
 name: Microservice-A-deployment
spec:
 replicas: 1
 selector:
   matchLabels:
     app: Microservice-A
 template:
   metadata:
     labels:
       app: Microservice-A
       date: 20thJune2021
     annotations:
       sidecar.istio.io/rewriteAppHTTPProbers: "false"
       proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
   spec:
     containers:
       - name: curl
         image: ewoutp/docker-nginx-curl
         imagePullPolicy: IfNotPresent
         command: [ 'sh', '-c', 'touch /tmp/healthy; echo The Pod is running && sleep 50' ]
         livenessProbe:
           exec:
             command:
               - cat
               - /tmp/healthy
           initialDelaySeconds: 15
           periodSeconds: 5
         lifecycle:
           postStart:
             exec:
               command: [ "/bin/sh", "-c", 'sleep 10;until [ $(eval curl -o -I -L -s -w "%{http_code}" http://microservice-B-api-service:9439/manage/health) -eq 200 ]; do echo "Waiting for microservice-B API";sleep 10;  done; exit 0' ]
       - name: Microservice-A
         image: microserviceA:latest
         imagePullPolicy: Always
         ports:[![enter image description here][1]][1]
           - name: port
             containerPort: 8080
         livenessProbe:
           httpGet:
             path: /actuator/health
             port: 8080
           initialDelaySeconds: 120
           periodSeconds: 30
           timeoutSeconds: 30
     imagePullSecrets:
       - name: dockersecret

Note: Reason for not using init-container: As we have implemented Istio with strict MTLS policy. https://github.com/istio/istio/issues/32039

Found below while searching for this issue on internet. enter image description here

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Ankita Sawant
  • 181
  • 2
  • 10
  • Hope the port of microservice is open and in same namespace please share more details regarding the service port config if possible or verifice one it's correct. – Harsh Manvar Jul 20 '21 at 12:23
  • 1
    service port is correct. I checked that. It keeps on restarting if you add a simple sleep statement in post restart hook as well. – Ankita Sawant Jul 20 '21 at 12:38
  • why using multiple containers ? ideally health you can also make part of existing single application only curl required inside the image and as soon as life cycle hook get executed and curl get 200 response you main container will start. – Harsh Manvar Jul 20 '21 at 12:41
  • @HarshManvar its not health check. We are finding alternate solution for init-container in case of Istio with strict MTLS – Ankita Sawant Jul 20 '21 at 12:45
  • if you want to start container after Micro service B get start....add your life cycle hook to micro service A container it will only start as soon as you get 200 res from micro service B. – Harsh Manvar Jul 20 '21 at 12:46

2 Answers2

1

That is because your command in postStart is sleeping for 10 seconds and your LivenessProbe is configured to fail after 5 seconds.

Maybe increase initialDelaySeconds or add a failureThreshold.

PSchoe
  • 66
  • 9
  • It's giving the same error after removing LivenessProbe and increasing initialDelaySeconds to 15 sec – Ankita Sawant Jul 20 '21 at 12:08
  • Hmm it seems that Kubernetes is executing containers in parallel as stated by [Paul linchpiner](https://linchpiner.github.io/k8s-multi-container-pods.html) (at the bottom of this doc). Init-Containers would execute in order and only on success, but you already said your environment does not allow for init-containers.. – PSchoe Jul 20 '21 at 12:36
1

You can use readinessProbe as well with the livenessProbe like this:

        readinessProbe:
            httpGet:
                path: /api/health
                port: 8080
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 8

        // for tcpSocket use:
            readinessProbe:
                tcpSocket:
                    port: 3306
KayV
  • 12,987
  • 11
  • 98
  • 148