5

I have a k8s cluster. Our service is queue based. Our pod subscribe to an event queue,fetch event and do tasks. So for this kind of service, how to define k8s liveness probe and readiness probe?

Bergi
  • 630,263
  • 148
  • 957
  • 1,375
Ruoxuan.Wang
  • 179
  • 1
  • 2
  • 13

2 Answers2

4

Following is a very brief introduction to these probes:

Liveliness Probe is for the Kubernetes to know if a workload is healthy. It could be a shell command executed in your container or a simple tcp/http request which should respond positively.

If a liveliness check fails after a period of timeout which is specified in the pod config, Kubrenetes will restart the workload.

So, if your workload is doing time consuming processes, you might need to give your liveliness probe enough time to make sure that your pod is not restarted unduely.

Rediness Probe is for the Kubernetes proxy to decide if your workload is ready for consuming traffic. The traffic will be sent to your pod only if the rediness probe responds positively. So, if your workload needs more time processing a single request and needs other requests to be diverted to other replicas for fast processing during this time, you might want to give a slightly high rediness interval to the workloads.

These probe parameters, combined with number of replicas can ensure fast and healthy functioning of your application. It is very important to understand the area each probe cover and the parameters you can tune them by.

Here are some reads:

https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Charlie
  • 22,886
  • 11
  • 59
  • 90
4

Assuming your problem is because is a processing worker consuming queue messages it doesn't expose any port to check.

In that case, you can define livenessProbe and readinessProbe custom command, next is an example from the documnetation:

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5
    readinessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

Also, take in mind the time your process takes to be live and ready to adjust the initialDelaySeconds and periodSeconds to not kill the pod before it is fully loeaded.

Peter
  • 1,658
  • 17
  • 23
wolmi
  • 1,659
  • 12
  • 25
  • So in this way, we need to create a thread to write timestamp log to file periodically. can liveness probe to check whether the log is stale. right? – Ruoxuan.Wang Jul 03 '20 at 09:16
  • With the command you can make any checks you need inside the pod, not only check files, you can check process status or other checks related to your worker. – wolmi Jul 03 '20 at 10:15
  • I am a little confused. What's the process status, How to check status related to worker? any example? Which command? – Ruoxuan.Wang Jul 06 '20 at 09:08
  • It depends totally on your worker code, the `command` livenessProbe allows you to execute any command or script you want to check the worker/process you are running. I cannot give you an example without seen your worker code. – wolmi Jul 06 '20 at 11:58