1

I'm using Airflow 1.9.0. And we realized that the scheduler get stuck sometimes waiting for a long running task to finish before starting a new set of tasks. When that happens all airflow tasks are defunct, except the long jobs. 

root      5111     1  0 17:58 ?        00:00:02 /usr/bin/python2 /bin/airflow scheduler -n 5 --pid /run/airflow/scheduler.pid
root      5489  5119  0 17:58 ?        00:00:11 /usr/bin/python2 /usr/bin/airflow run ....
root      5522  5489  0 17:59 ?        00:00:01 /usr/bin/python2 /usr/bin/airflow run ....
root      5118  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5119  5111  0 17:58 ?        00:00:00 /usr/bin/python2 /bin/airflow scheduler -n 5 --pid /run/airflow/scheduler.pid
root      5120  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5121  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5122  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5123  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5124  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5125  5111  0 17:58 ?        00:00:00 [airflow] <defunct>
root      5126  5111  0 17:58 ?        00:00:00 [airflow] <defunct>

Any idea what we're doing wrong?

Thanks,

Younes
  • 375
  • 4
  • 16
  • With reference to `..all airflow schedulers are defunct..`; by any chance did you mean `all airflow tasks are defunct`? because running **multiple `scheduler`s** [doesn't sound like a good idea](https://stackoverflow.com/a/39595535/3679900) – y2k-shubham Feb 01 '19 at 18:36
  • You right of course! I meant all tasks... – Younes Feb 01 '19 at 19:45

1 Answers1

3

Remove the -n 5 from /bin/airflow scheduler -n 5. The -n is a legacy airflow feature. The LocalExecutor runs in the process, hence this behavior. Use Airflow 1.8.1+ and do not use "-n 5" and you will not encounter the issue with LocalExecutor.

For more information on it: https://issues.apache.org/jira/browse/AIRFLOW-401?focusedCommentId=16094732&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16094732

kaxil
  • 17,706
  • 2
  • 59
  • 78