3

Airflow stopped running tasks all of a sudden. Below are all running airflow scheduler airflow webserver airflow worker

webui message

All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless: - The scheduler is down or under heavy load

If this task instance does not start soon please contact your Airflow administrator for assistance.

Scheduler seems to be in a loop, keeps repeating the below messages. WebUI shows tasks are in queued state. Tried restarting the scheduler, didn't help.

[2018-11-17 22:03:45,809] {{jobs.py:1607}} DEBUG - Starting Loop...
[2018-11-17 22:03:45,809] {{jobs.py:1627}} INFO - Heartbeating the process manager
[2018-11-17 22:03:45,810] {{jobs.py:1662}} INFO - Heartbeating the executor
[2018-11-17 22:03:45,810] {{base_executor.py:103}} DEBUG - 124 running task instances
[2018-11-17 22:03:45,810] {{base_executor.py:104}} DEBUG - 0 in queue
[2018-11-17 22:03:45,810] {{base_executor.py:105}} DEBUG - 76 open slots
[2018-11-17 22:03:45,810] {{base_executor.py:132}} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
[2018-11-17 22:03:45,810] {{celery_executor.py:80}} DEBUG - Inquiring about 124 celery task(s)

Airflow setup: apache-airflow[celery, redis, all]==1.9.0

I also checked these posts but didn't help me:

Airflow 1.9.0 is queuing but not launching tasks

Airflow tasks get stuck at "queued" status and never gets running

moon
  • 1,702
  • 3
  • 19
  • 35

1 Answers1

4

Problem solved. This is a problem when you create your build on or after 2018-11-15 Turns out apache-airflow[celery, redis, all]==1.9.0 takes the latest version of redis-py 3.0.1 which does not work with celery 4.2.1. Solution is to use redis-py 2.10.6 redis==2.10.6 apache-airflow[celery, all]==1.9.0

moon
  • 1,702
  • 3
  • 19
  • 35