Tell when Job is Complete

Question

I'm looking for a way to tell (from within a script) when a Kubernetes Job has completed. I want to then get the logs out of the containers and perform cleanup.

What would be a good way to do this? Would the best way be to run kubectl describe job <job_name> and grep for 1 Succeeded or something of the sort?

Answer 1

Since version 1.11, you can do:

kubectl wait --for=condition=complete job/myjob

and you can also set a timeout:

kubectl wait --for=condition=complete --timeout=30s job/myjob

Answer 2

You can visually watch a job's status with this command:

kubectl get jobs myjob -w

The -w option watches for changes. You are looking for the SUCCESSFUL column to show 1.

For waiting in a shell script, I'd use this command:

until kubectl get jobs myjob -o jsonpath='{.status.conditions[? 
    (@.type=="Complete")].status}' | grep True ; do sleep 1 ; done

Answer 3

You can use official Python kubernetes-client.

https://github.com/kubernetes-client/python

Create new Python virtualenv:

virtualenv -p python3 kubernetes_venv activate it with

source kubernetes_venv/bin/activate

and install kubernetes client with:

pip install kubernetes

Create new Python script and run:

from kubernetes import client, config

config.load_kube_config()

v1 = client.BatchV1Api()
ret = v1.list_namespaced_job(namespace='<YOUR-JOB-NAMESPACE>', watch=False)
for i in ret.items:
    print(i.status.succeeded)

Remember to set up your specific kubeconfig in ~/.kube/config and valid value for your job namespace -> '<YOUR-JOB-NAMESPACE>'

Answer 4

I would use -w or --watch:

$ kubectl get jobs.batch --watch
NAME     COMPLETIONS   DURATION   AGE
python   0/1           3m4s       3m4s

Answer 5

Adding the best answer, from a comment by @Coo, If you add a -f or --follow option when getting logs, it'll keep tailing the log and terminate when the job completes or fails. The $# status code is even non-zero when the job fails.

kubectl logs -l job-name=myjob --follow

One downside of this approach, that I'm aware of, is that there's no timeout option.

Another downside is the logs call may fail while the pod is in Pending (while the containers are being started). You can fix this by waiting for the pod:

# Wait for pod to be available; logs will fail if the pod is "Pending"
while [[ "$(kubectl get pod -l job-name=myjob -o json | jq -rc '.items | .[].status.phase')" == 'Pending' ]]; do
    # Avoid flooding k8s with polls (seconds)
    sleep 0.25
done

# Tail logs
kubectl logs -l job-name=myjob --tail=400 -f

Answer 6

Although kubectl wait --for=condition=complete job/myjob and kubectl wait --for=condition=complete job/myjob allow us to check whether the job completed but there is no way to check if the job just finished executing (irrespective of success or failure). If this is what you are looking for, a simple bash while loop with kubectl status check did the trick for me.

#!/bin/bash
while true; do
status=$(kubectl get job jobname -o jsonpath='{.status.conditions[0].type}')
echo "$status" | grep -qi 'Complete' && echo "0" && exit 0
echo "$status" | grep -qi 'Failed' && echo "1" && exit 1
done

Answer 7

It either one of these queries with kubectl

kubectl get job test-job -o jsonpath='{.status.succeeded}'

or

kubectl get job test-job -o jsonpath='{.status.conditions[?(@.type=="Complete")].status}'