Airflow marks dataflow task as failed but it actually succeeded

Asked Jun 16 '20 at 09:09

Active Jun 16 '20 at 09:09

Viewed 688 times

I am using gcloud.

With Composer I have an airflow environment that schedules diffferent task pipelines.

One of those pipelines consist in:

few tasks to prepare for the dataflow job
The Dataflow job
Some tasks that will exploit the results of the jobs

The DAG is configured for each step to wait for the former ones to have succeeded.

I have an issue, the dataflow task is marked as failed in airflow, so the next steps are not executed. However when I check the job in dataflow it seems to have succeeded.

this is all the (cut) log i have in airflow:

Is this a known issue ?

asked Jun 16 '20 at 09:09

Louis D.

Please take a look at this : https://stackoverflow.com/questions/61919610/google-cloud-composerairflow-dataflow-job-inside-a-dag-executes-successfully – Priya Agarwal Jun 17 '20 at 05:31
Could you share the full log message where it mentions the job fails? – Alexandre Moraes Jun 17 '20 at 09:08
that is all i have in the logs, I see it failed in the tree representation (red square) – Louis D. Jun 18 '20 at 11:08
Please take a look on the link provided by @PriyaAgarwal. I just posted my implementation of a workaround there. – itroulli Jul 23 '20 at 16:12

Airflow marks dataflow task as failed but it actually succeeded

0 Answers0