7

I have a situation in which I run a Kubernetes Job and then want to delete the pod in which the Job's containers run when the job completes. The Kubernetes documentation says the it is the responsibility of the user to delete the pod.

However, my goal is to run a CPU intensive job by spinning up a big honking pod, let the job run, and then upon job completion, automagically nuke the pod. My thinking is that paying for an expensive, resource intensive pod is worth it when the revenue producing job runs. Once the job is over, the revenue stream ends and I need to cut the expense, hence nuking the pod.

I am thinking of having my code in the job container send a out a message upon program completion to a publisher that is picked up by a subscriber. The subscriber knows how to run the command, kubectl delete jobs/myContainer to nuke the pod.

But, maybe there is a better way. Thus, I ask.

Thanks in advance.

reselbob
  • 365
  • 3
  • 13
  • 1
    Two things: 1, this question was asked before ([the link in the Related column](https://stackoverflow.com/questions/36384873/kubernetes-job-cleanup?rq=1) was handy, but not the only one), and 2, every Pod has a ServiceAccount and thus can delete itself if it feels it has exited cleanly – mdaniel Dec 06 '17 at 06:35
  • 1
    Thank you @MatthewLDaniel. I will be more attentive moving forward to doing better research before posing a question. – reselbob Dec 09 '17 at 23:31

2 Answers2

4

There is a new API that delete jobs after completion and all the dependent elements. The TTLAfterFinished.

You can define it in the .spec.ttlSecondsAfterFinished property of your job. The cleanup will happen X seconds after the job is completed or failed.

As of 16, March, 2019, this API is in alpha, with version 1.12 and can only be used with alpha features enabled.

Navid Khan
  • 979
  • 11
  • 24
Benoît Latinier
  • 2,062
  • 2
  • 24
  • 36
  • 35min after a 100 seconds ttlSecondsAfterFinished, the job is done and deleted, but the pod is still active with a status of `completed` – Pian0_M4n Apr 20 '22 at 17:54
3

Why would you need to delete the pod at all ? If the job completed successfully (exit code 0) then the pod will be completed, it will no longer use cpu/memory resurces, and should be garbage collected when needed.

Radek 'Goblin' Pieczonka
  • 21,554
  • 7
  • 52
  • 48
  • 3
    Thank you, Radek. It is unclear from the documentation that the Pod is subject to GC. From the doc: _When a Job completes, no more Pods are created, but the Pods are not deleted either. Since they are terminated, they don’t show up with kubectl get pods, but they will show up with kubectl get pods -a."_ (https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-termination-and-cleanup When I ran a proof of concept locally under minikube I saw that the pod still persisted. I wonder do you know the cleanup period on the GC. Again, THANKS! – reselbob Dec 09 '17 at 23:27
  • I agree with @reselbob, the precise line he has cited is particularly unclear. – franklin May 05 '18 at 07:07
  • I would second @reselbob and want to add a data point, in my case the resource was not released until 4-5 minutes after job completion. At the time when job completes, the pod occupies 547MB, then it keeps occupying that much memory for the next 4-5 minutes. This is pretty troublesome for me as I'm processing a pool of jobs while wanting to limit concurrency of 2 jobs. But this lag GC is causing memory spike and could blown up the entire node. – Shawn May 17 '20 at 22:49