I want to set-up liveness and readiness probes for Celery worker pods. Since these worker pods doesn't have a specific port associated to them I am finding it difficult. Main Django app nginx server was easier to set-up.
I am very new to k8s so not much familiar to the different ways to do it.
liveness probe for celery worker: This command only works when remote control is enabled.
$ celery inspect ping -d <worker_name> --timeout=<timeout_time>
When a celery worker uses a solo pool, healthcheck waits for the task to finish. In this case, you must increase the timeout waiting for a response.
so in yaml:
livenessProbe:
initialDelaySeconds: 45
periodSeconds: 60
timeoutSeconds: <timeout_time>
exec:
command:
- "/bin/bash"
- "-c"
- "celery inspect ping -d <worker_name> | grep -q OK"
Of course you have to change the worker name and timeout to your own values
Basically, configuration files in YAML for Celery liveness and readiness probes can be set up in a general way as described in Kubernetes docs. You can specifically adjust them for Celery pods based on your requirements, for example:
livenessProbe:
exec:
# bash is needed to replace the environment variable
command: [
"bash",
"-c",
"celery inspect ping -A apps -d celery@$HOSTNAME"
]
initialDelaySeconds: 30
periodSeconds: 60
timeoutSeconds: 10
There is still ongoing GitHub issue on this matter - other solutions that might be useful for your set-up are described here.
