Workflow orchestration for Google Dataflow

Question

We are using Google Dataflow for batch data processing and looking for some options for workflow orchestration tools something similar to what Azkaban does for Hadoop.

Key things things that we are looking for are,

Configuring workflows
Scheduling workflows
Monitoring and alerting failed workflows
Ability to rerun failed jobs

We have evaluated Pentaho, but these features are available in their Enterprise edition which is expensive. We are currently evaluating Azkaban as it supports javaprocess job types. But Azkaban is primarily created for Hadoop jobs so it has more deep integration with Hadoop infrastructure then plain javaprocesses.

Appreciate some suggestions for opensource or very low cost solutions.

score 1 · Answer 1 · answered Aug 18 '16 at 05:11

1

It sounds like Apache Airflow (https://github.com/apache/incubator-airflow) should meet your needs and it now has a Dataflow operator (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/dataflow_operator.py).

answered Aug 18 '16 at 05:11

William Vambenepe

140
4

Thanks William, we have started evaluating Airflow to see if it worksout. – Mayur Shah Aug 24 '16 at 06:32
@user73827 how did the evaluation go? – Andrew Jun 19 '17 at 20:49
any comments about Airflow? I'm very interested. – Mario Sep 25 '17 at 17:00
I am struggling to Orchestrate Dataflow jobs with Cloud Composer. Don't go for it. I tried 10 different approaches and it does't work – Anurag Yadav May 04 '20 at 06:55

score 0 · Answer 2 · answered Aug 03 '18 at 04:48

To orchestrate the Google dataflow we can use Cloud composer which is managed workflow orchestration service built on Apache Airflow. It gives more flexibility, using this we can orchestrate most of the google services and workflows that cross between on-premises and the public cloud.

Workflow orchestration for Google Dataflow

2 Answers2

Linked