Kubernetes CronJob with a sidecar container

Question

I am having some issues with a Kubernetes CronJob running two containers inside of a GKE cluster.

One of the two containers is actually executing the job that must be done by the CronJob.

This works perfectly fine. It is started when it is supposed to be started, does the job and then terminates. All fine up until this point.

What seems to be causing some issues is the second container, which is a sidecar container used to access a database instance. This won't terminate and seems to be leading to the problem that the CronJob itself won't terminate. Which is an issue, since I see an accumulation of running Job instances over time.

Is there a way to configure a Kubernetes batch CronJob to be terminating when one of the container is successfully exe

apiVersion: batch/v1
kind: CronJob
metadata:
  name: chron-job-with-a-sidecar
  namespace: my-namespace
spec:
#            ┌───────────── minute (0 - 59)
#            │ ┌───────────── hour (0 - 23)
#            │ │  ┌───────────── day of the month (1 - 31)
#            │ │  │ ┌───────────── month (1 - 12)
#            │ │  │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
#            │ │  │ │ │                                   7 is also Sunday on some systems)
#            │ │  │ │ │                                   OR sun, mon, tue, wed, thu, fri, sat
#            │ │  │ │ │
  schedule: "0 8 * * *" # -> Every day At 8AM
  jobTemplate:
    metadata:
      labels:
        app: my-label
    spec:
      template:
          containers:
          # --- JOB CONTAINER -----------------------------------------------
          - image: my-job-image:latest
            imagePullPolicy: Always
            name: my-job
            command:
            - /bin/sh
            - -c
            - /some-script.sh; exit 0;
          # --- SIDECAR CONTAINER ----------------------------------------------
          - command:
            - "/cloud_sql_proxy"
            - "-instances=my-instance:antarctica-south-3:user=tcp:1234"
            # ... some other settings ...
            image: gcr.io/cloudsql-docker/gce-proxy:1.30.0
            imagePullPolicy: Always
            name: cloudsql-proxy
            # ... some other values ...

Mafor · Accepted Answer · 2022-11-19T11:33:46.277

No, strictly speaking there is no way to make Kubernetes stop a sidecar container automatically once a "main" container is done.

The closest "kubernetes-native" solution I can think of is setting CronJob concurrencyPolicy to Replace (see CronJobSpec). It won't stop the sidecar once done, but at least each new job will be overriding the previous one, so the jobs won't be accumulating. Unfortunately, with this solution, you are gonna lose job history.

If this solution does not fit your needs, you will need to implement some form of communication between containers, but nothing like that is built in into Kubernetes itself. There are some external tools though, that can help, e.g. kubexit.

This [Answer](https://stackoverflow.com/a/72693205/696508) was a lot less convoluted imo. — George, Feb 13 '23 at 12:17

score 2 · Answer 2 · answered Mar 16 '23 at 12:51

This is the way:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: sidecar-example
spec:
  schedule: '0 0 * * 5'
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          shareProcessNamespace: true
          containers:
            - name: myjob
              image: myjob:latest
              command: ['/bin/sh', '-c']
              args:
                - |
                  do-cron-job.sh; \
                  killall -SIGTERM sidecar-process
                  
            - name: sideacr
              image: mysidecar:latest

Use shareProcessNamespace: true and kill the main process of the sidecar container on exit.

And let's way for the https://kubernetes.io/docs/tasks/job/pod-failure-policy/ feature to become stable. It will help with the non 0 exit codes of the sidecar container.

score 1 · Answer 3 · edited Jun 05 '23 at 17:57

1

KEP-753 will add support for terminating jobs/pods that have sidecar containers.

This feature is not released yet. See this issue to track the process.

edited Jun 05 '23 at 17:57

coherence73

3
2

answered Mar 27 '23 at 18:09

josescuderoh

65
10

You know about the commenting privilege which you do not have, so well that you can even put it into words. You are aware of the rule https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead . In that situation please do not decide to misuse a different mechanism (an answer) for something it is not meant for and which you are not allowed yet to do. – Yunnosch Mar 27 '23 at 18:46
So, if you mean this to be a comment, please delete it. But consider whether you can [edit] this into an answer according to [answer]. – Yunnosch Mar 27 '23 at 18:47

Kubernetes CronJob with a sidecar container

3 Answers3