kubernetes get events for finished job

Question

A cronjob we scheduled at night was started correctly but the image itself run much later than scheduled. We expect that there was a problem either pulling the image or requesting the resources from the cluster. Usually I can see such errors in the events section of the kubectl describe job-name output. In this section I can see events such as pull image create container etc. But after the job is finished, no events are shown anymore.

Is it possible the see those events for a finished job?

Or is there another way to investigate such a problem?

Maybe this can help you: https://stackoverflow.com/questions/53371505/access-logs-in-cron-jobs-kubernetes — Arslanbekov Denis, Apr 23 '19 at 08:32

score 5 · Accepted Answer · answered Apr 23 '19 at 08:41

The problem with storing events is wider than just cronjobs. Events in Kubernetes by default are stored only for 1 hour (--event-ttl flag for kube-apiserver). That means that if your cronjob was run two hours ago - you will not see events in "kubectl describe".

In order to save events for later investigations you need to export them somewhere. For example, Google Kubernetes Engine stores events into Stackdriver. For vanilla Kubernetes you may store events in Prometheus using event_exporter or to Elasticsearch. Does that answer your question?

This is indeed very helpful information. Thank you. – mjspier Apr 23 '19 at 08:55 — mjspier, Apr 23 '19 at 08:55

kubernetes get events for finished job

1 Answers1