2

We have a kind of evaluation job which consists of several thousand invocations of a legacy binary with various inputs, each of which running like a minute. The individual runs are perfectly parallelizable (one instance per core).

What is the state of the art to do this in a hybrid cloud scenario?

Kubernetes itself does not seem to provide an interface for prioritizing or managing waiting jobs. Jenkins would be good at these points, but feels like a hack. Of course, we could hack something ourselves, but the problem should be sufficiently generic to already have an out-of-the box solution.

Gunther Vogel
  • 133
  • 1
  • 7
  • Please, upvote/accept the answer to let community know about the usefulness, thanks! – aga Nov 07 '19 at 12:44

2 Answers2

1

There are a lot of frameworks that helps managing jobs in Kubernetes cluster. The most popular are:

  • Argo for orchestrating parallel jobs on Kubernetes. Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).
  • Airflow - has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Also take a look for kubernetes-executor.

I recommend you to look for this video which describe each of framework and help you decide which is better for you.

aga
  • 3,790
  • 3
  • 11
  • 18
  • Thank you for your recommendations. We ended up implementing a custom controller, owed mainly to the fact that it would have been hard to make the legacy binary satisfy the resilience requirements. – Gunther Vogel Jan 21 '20 at 08:25
1

You may be interested in following aricles about using Mesos for Hybrid Cloud

Hybrid cloud technology is becoming increasingly popular as it merges private and public clouds to bring the best of two worlds together. However, due to the heterogeneous cloud installation, facilitating a hybrid cloud setup is not simple. Despite the availability of some commercial solutions to build a hybrid cloud, an open source implementation is still unavailable. In this paper, we try to bridge the gap by providing an open source implementation by leveraging the power of Apache Mesos. We build a hybrid cloud on the top of multiple cloud platforms, private and public.

janisz
  • 6,292
  • 4
  • 37
  • 70