With Airflow 1.8.1, I am using LocalExecutor with max_active_runs_per_dag=16, I called a for loop to dynamic create tasks (~100) with PythonOperators. Most time the tasks completed without any issues. However, it is still possible that task is with queue status but scheduler seems forget it, I can clear the task and able to rerun the queued task and worked, but would like to know how to avoid stuck in queue.
Asked
Active
Viewed 551 times
0
Chengzhi
- 2,531
- 2
- 27
- 41
-
Could you share some log output and source code if possible? That'll make it easier to help out. – William Hammond Aug 20 '17 at 14:11
-
What is the `parallelism` config of your LocalExecutor? This one defines the number of running processes. – Fokko Driesprong Aug 29 '17 at 13:52
-
@FokkoDriesprong parallelism = 32 – Chengzhi Aug 29 '17 at 14:31
-
This is due to a issue with 1.8.1 as mentioned here and should be fixed at 1.9 https://stackoverflow.com/a/46878502/1451653 – Chengzhi Jan 23 '18 at 13:52