0

I used airflow for some scheduled work and recently, I found that the task in my dag takes lots to time to turn the status from scheduled to running and from running to succes.

Task took around 27 seconds to run

But when it is success, it shown 2.6 seconds

Task took around 27 seconds to run, but when it is succeed, it shown 2.6 seconds to run.

How is that happening for my airflow?

  • Version: v2.1.2

    Git Version: .release:2.1.2+d25854dd413aa68ea70fb1ade7fe01425f456192

    executor: CeleryExecutor

Some configuration of my airflow settings:

  • worker_concurrency = 24
  • parallelism = 16
  • dag_concurrency = 12
  • max_active_runs_per_dag = 32
wayne64001
  • 399
  • 1
  • 3
  • 13
  • We had the same issue with airflow 1.X series. We played with some settings, following recommendations [in this other SO thread](https://stackoverflow.com/q/47450855) but we still couldn't improve much. Maybe you'll have better luck than us. – Savir Mar 28 '23 at 20:30

0 Answers0