Deadlock with ExternalTaskOperators: Why, how often, and how to get around that problem?

Question

I am implementing scheduled pipelines and currently I am using an ExternalTaskOperator to set inter-DAG dependency. I read here that if you don't manually raise the priority of the upstream tasks it's possible that there will be deadlock.

I was wondering how common this situation is, how you manually raise priority levels of different tasks (the source code of many, like Bash and Python Operators, don't seem to have a priority_level param), and if there are any other better methods of setting inter-DAG dependencies.

Thanks

Answer 1

I've never used ExternalTaskSensor in production, so can't comment on how often deadlocks occur. But apart from priority_weight / weight_rule that you already mentioned, I can think of 2 more ways to try to overcome this

• Using Airflow pools to guarantee dedicated slots for sets tasks
• Using mode param of Sensors (BaseSensorOperator)