Do Kubernetes pods still receive requests after receiving SIGTERM?

Question

I want to implement graceful shutdown in a Kubernetes Pod. I know I need to listen for SIGTERM, which indicates the start of the shutdown procedure. But what exactly do I do when I receive it?

At the very least I must wait for all running request to finish before exiting. But can the pod still receive new requests after receiving the SIGTERM? (It's exposed using a service.) I can't find any explicit documentation on this.

The docs state:

Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly can continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.

So that seems to imply that new requests can still come in. So how long should I continue to expect new requests before graceful termination? Do I simply ignore the SIGTERM, continue to serve requests as usual and wait for the eventual SIGKILL?

I suppose ensuring future readiness checks fail and then waiting longer than the period with which they occur before terminating might work?

I'm on Kubernetes 1.2.5, if that makes any difference, and am talking about rolling updates in particular, but also scaling replication controllers down generally.

score 3 · Answer 1 · answered Nov 15 '17 at 23:46

I recently faced similar problem, I used simple preStop hook, which introduces some delay(sleep) between start of termination and receiving SIGTERM to underlying process

lifecycle:
        preStop:
          exec:
            command:
              - "sleep"
              - "60"

This delay helps,

Load balancer to remove(sync) the pod being terminated
Gives chance to terminating pod to complete requests received before termination
Fulfill the requests received by terminating pod between termination and load balancer update(sync)

PreStop can be made more intelligent for pod with unpredictable time of serving

score 1 · Answer 2 · answered Nov 16 '16 at 06:03

You should use the preStop hook along with a livenessProbe health check if you want to cleanly drain traffic before shutting down the pod.

Ideally, what you would have is a preStop hook that forces the pod into an unhealthy livenessProbe check so the pod will be removed from the load balancer and then gracefully shuts down.

This isn't pretty but the example worked in my simple tests.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        livenessProbe:
          exec:
            command:
            - cat
            - /usr/share/nginx/html/50x.html
          initialDelaySeconds: 15
          timeoutSeconds: 1
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              # SIGTERM triggers a quick exit; fail health check and gracefully terminate instead
              command: ["/bin/rm","-f","/usr/share/nginx/html/50x.html",";","sleep","2",";","/usr/sbin/nginx","-s","quit"]

From this example the livenessProbe looks for the /usr/share/nginx/html/50x.html file. As long as that file exists the pod is healthy. When the pod is going to be shut down the preStop hook is fired which removes that file. This should trigger the pod to be removed from an external load balancer on the next health check (1 sec). The preStop command then sleeps 2 seconds (to make sure the next health check fired) and tells nginx to gracefully stop -s quiet. The preStop command is supposed to complete within 30 seconds before the pod is force killed (SIGTERM) but that should give plenty of time for nginx to drain connections.

Thanks, but won't a failing liveness probe immediately kill the container, closing any open connections? A readiness probe seems more appropriate to me. Either way, what's the difference between using a preStop hook and catching SIGTERM? — Mr. Wonko, Nov 16 '16 at 13:52
If a pod fails liveness it will go through the normal termination procedure. If a pod is already being killed it won't be re-killed unless it exceeds the 30 second timeout (default time). Readiness probes are only used at startup to make sure the pod is ready before sending traffic. Once a pod is ready the readiness probe is no longer checked and future checks use livenessProbe. PreStop lets you reuse generic containers and do custom actions per pod instead of building a specialized container per application/pod to handle clean shutdowns. — Justin Garrison, Nov 16 '16 at 16:57
I did some experiments and the readiness probe is in fact periodically checked and if a pod ceases to be ready it will no longer receive traffic. — Mr. Wonko, Nov 24 '16 at 17:59

score 1 · Accepted Answer · answered Nov 28 '16 at 10:56

I ran some experiments to find out exactly what happens.

The pod will briefly (<1s) continue receiving requests after shutdown is initiated, so you need to either catch SIGTERM or install a preStop hook so you can wait for them (and finish serving current requests).

However, once the shutdown has been initiated, the readiness probe no longer matters, you don't need to change its state to stop receiving requests. (But before that a failing readiness probe will lead to your pod receiving no more traffic.)

Do Kubernetes pods still receive requests after receiving SIGTERM?

3 Answers3

Linked