11

If I want to run multiple replicas of some container that requires a one off initialisation task, is there a standard or recommended practice?

Possibilities:

  • Use a StatefulSet even if it isn't necessary after initialisation, and have init containers which check to see if they are on the first pod in the set and do nothing otherwise. (If a StatefulSet is needed for other reasons anyway, this is almost certainly the simplest answer.)
  • Use init containers which use leader election or some similar method to pick only one of them to do the initialisation.
  • Use init containers, and make sure that multiple copies can safely run in parallel. Probably ideal, but not always simple to arrange. (Especially in the case where a pod fails randomly during a rolling update, and a replacement old pod runs its init at the same time as a new pod is being started.)
  • Use a separate Job (or a separate Deployment) with a single replica. Might make the initialisation easy, but makes managing the dependencies between it and the main containers in a CI/CD pipeline harder (we're not using Helm, but this would be something roughly comparable to a post-install/post-upgrade hook).
armb
  • 278
  • 2
  • 13
  • 1
    If you are using `helm` to deploy, the best approach is to create a Job using [Helm Hooks](https://github.com/helm/helm/blob/master/docs/charts_hooks.md) feature. You can use `pre-install` and `pre-upgrade` hooks alongside with the `--wait` flag; this way Helm ensures that your job is completed before deploying your pods (or even aborting/failing the release if the job fails). – Eduardo Baitello Aug 29 '19 at 18:34
  • I'm not sure if it's an exact dupe but [I've answered before](https://stackoverflow.com/a/57388017/1318694) describing what @EduardoBaitello suggests. – Matt Aug 30 '19 at 04:03
  • Possible duplicate of [Kubernetes: Tasks that need to be done once per cluster or per statefulset or replicaset](https://stackoverflow.com/questions/57386429/kubernetes-tasks-that-need-to-be-done-once-per-cluster-or-per-statefulset-or-re) – Matt Aug 30 '19 at 04:05
  • It's also related to https://stackoverflow.com/questions/53427789/kubernetes-deployment-initialization-how-to-ensure-it-happens-only-once. I'm not sure why I didn't find that in my earlier search. Both have fairly limited answers in terms of describing trade-offs of different methods though. – armb Aug 30 '19 at 10:27
  • CockroachDB is an example of a Helm chart using an initialisation Job: https://github.com/helm/charts/tree/master/stable/cockroachdb They have instructions for a pure Kubernetes setup running a Job manually too - https://www.cockroachlabs.com/docs/stable/orchestrate-cockroachdb-with-kubernetes.html#manual It's integrating Jobs cleanly into a declarative GitOps CI/CD pipeline that's the non-obvious bit of that route. – armb Aug 30 '19 at 14:34
  • I can't remember what version was current at the time of my 2019 comment, but https://www.cockroachlabs.com/docs/v22.1/deploy-cockroachdb-with-kubernetes.html?filters=manual uses a different approach (and uses StatefulSets). – armb Aug 04 '22 at 15:52

2 Answers2

1

The fact that "replicas of some container" are dependent on "a one off initialisation task" means that the application architecture does not fit the Kubernetes paradigm well. That is why an involvement of a third-party manager on top of k8s like Helm has to be considered (as suggested by Eduardo Baitello and Matt).

To keep with pure Kubernetes approach, it'd be better to redesign your application so that get it components working as independent or loosely coupled microservices (including initialization tasks). A similar question has been discussed here recently.

As for the possibilities listed in the question, perhaps the first option with InitContainers and StatefulSets could be feasible in pure Kubernetes.

mebius99
  • 2,495
  • 1
  • 5
  • 9
  • 1
    Redesigning the application to not use any third party database or other component with the same pattern is not an option :-) I'm currently considering a pure Kubernetes solution where the work is done in a Job, which all the init containers try to create, but where at most one of them will succeed, effectively using the Kubernetes API as a mutex. The init containers that don't create a Job because it already exists will block until the existing one completes (and do nothing else). – armb Sep 03 '19 at 10:00
  • Accordingly with Kubernetes good practices you should redesign your application so that each application replica worked with its own database replica. That way you'll get independent pods each of which could be easily initialized with its own InitContainer. – mebius99 Sep 03 '19 at 13:25
  • 10
    that rather misses the point of having a database. It's true that Kubernetes used to be considered only suitable for stateless applications, but things have moved on. – armb Sep 03 '19 at 15:11
1

We effectively ended up with a Job that does the initialization task and creates a secret that the Deployment replicas have mounted as a volume, blocking them until the Job has completed. We're using ArgoCD without sync waves. (There are complications with patching the Job name whenever its spec is updated because Jobs are immutable, but they aren't directly relevant to the original question.)

armb
  • 278
  • 2
  • 13