2

I am a beginner in docker，and I want to run a pipeline in a cluster server with docker.

The pipeline flow chart is:

pipeline flow chart

the Task_A is the input of Task_B_1 2 3, and the all outputs of Task_B_1 2 3 is the input of Tast_C.

Task_B_1 2 3 are parallel.

The container will stop after finish the task.

Every Task is a docker container.

I want to know whether a docker cluster manage tool (such as Mesos、Chronos、Kubernetes or other) can manage and monitor the running of the pipeline.

Busti
  • 5,492
  • 2
  • 21
  • 34
Kang Li
  • 109
  • 9
  • i'm not sure but have a look to docker-swarm – bxN5 Aug 19 '18 at 10:52
  • How would you do this without Docker? How does data flow between the tasks? – David Maze Aug 19 '18 at 10:59
  • "I want to know whether a docker cluster manage tool ..." I'm pretty sure they all can, but I'm only sure for kubernetes. But if your only goal is running the task in parallel, you shouldn't use cluster-orchestration. Maybe look at spark, snakemake, etc (which work well with in orchestrated clusters as well) – wotanii Aug 19 '18 at 12:07
  • 4
    You may want to checkout Argo - The workflow engine for kubernetes (https://github.com/argoproj/argo/blob/master/README.md) – Bal Chua Aug 19 '18 at 13:14

1 Answers1

0

the Common Workflow Language (CWL),such as nextcloud, snakemake, cromwell, can solve the problem.

Kang Li
  • 109
  • 9