3

I have a problem with long waits between tasks in the same DAG. I'm running Airflow 1.10. Can anyone point to which settings that are relevant to tweak? Please see example below.

Airflow tasks

The longest wait here is more than one and a half hour.

Wessi
  • 1,702
  • 4
  • 36
  • 69
  • Just dropping another data point here but I am sometimes seeing up to ~30 min latency between tasks. Several minutes is standard in my cluster, which is managed cloud composer on google cloud. I haven't found any solution, but other people do seem to have this issue. Seems hard to believe that it could perform so poorly. FWIW, one of the main improvements of airflow 2 is that the speed to schedule tasks is greatly reduced. – Hippocrates Jan 22 '21 at 16:12

1 Answers1

0

Only thing that worked for me is restarting the scheduler. So I now have a cron job to restart the scheduler every 5 minutes.

Wessi
  • 1,702
  • 4
  • 36
  • 69