2

Running on Kubernetes v1.20

I have a startup probe configured and a liveness probe. On the first start of the container, the startup probe is executed until the liveness probe takes over (as documented). However, it seems if the liveness probe fails and the container restarts, the startup probe is not executed again. Is this intended behavior? I cannot find this documented anywhere.

To reproduce this issue, I'm running the following container definition (relevant parts only):

containers:
      - args:
        - /bin/sh
        - -c
        - touch /tmp/alive; sleep 10000
        image: busybox
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - touch /tmp/liveness; test -f /tmp/alive
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 2
        startupProbe:
          exec:
            command:
            - touch
            - /tmp/startup
          failureThreshold: 3
          periodSeconds: 2
          successThreshold: 1
          timeoutSeconds: 2

So if the liveness probe runs, it creates /tmp/liveness. If the startup probe runs, it creates /tmp/startup. You can simulate the liveness check failing by deleting /tmp/alive.

On first startup:

$ ls /tmp/
alive     liveness  startup

After rm /tmp/alive, the liveness check fails and the container is restarted. Then, in the new container:

$ ls /tmp/
alive     liveness

So it seems the startup probe is not executed anymore.

  • might want to check if your K8 version is the one affected by this issue: https://github.com/kubernetes/kubernetes/issues/101064 or https://github.com/kubernetes/kubernetes/issues/102230 – Raymond Hartoyo Nov 29 '21 at 13:56
  • @Raymond Yup, we are on 1.20.6. Seems like my Google-fu was not good enough when looking for known issues. Will re-test after an update but that's probably it. Feel free to post it as an answer so I can give credit where credit is due. – Jens De Temmerman Nov 29 '21 at 14:08
  • ok, posted as an answer, in case anyone experiencing same thing – Raymond Hartoyo Nov 29 '21 at 14:42
  • I'm experiencing the same issue on v1.19.10, which I thought was the patched one – David Lukac Feb 14 '22 at 13:05

1 Answers1

3

Might want to check if your K8 version is the one affected by this issue:

https://github.com/kubernetes/kubernetes/issues/101064

or

https://github.com/kubernetes/kubernetes/issues/102230

Raymond Hartoyo
  • 266
  • 1
  • 6