6

I'am having an issue with airflow when running it on a 24xlarge machine on EC2.

I must note that the parallelism level is 256.

For some days the dagrun finishes with status 'failed' for two undetermined reasons :

  1. Some task has the status 'upstream_failed', which is not true because we can see clearly that all the previous steps where successful. enter image description here

  2. Other tasks have not the status 'null', they have not started yet and they cause the dagrun to fail. enter image description here

I must note that the logs for both of these tasks are empty

enter image description here

And here is the tast instance details for these cases :

enter image description here

Any solutions please ?

I.Chorfi
  • 507
  • 2
  • 5
  • 12

2 Answers2

3

The other case where I've experienced the second condition ("Other tasks have not the status 'null'"), is when the task instance has changed, and specifically changed operator type.

I'm hoping you already got an answer / were able to move on. I've been stuck on this issue a few times in the last month, so I figured I would document what I ended up doing to solve the issue.

Example:

  • Task Instance originally is an instance of a SubDag Operator
  • Requirements cause the type of the operator to change from a SubDag Operator to a Python Operator
  • After the change, the Python Operator is set to state NULL

As best I can piece together, what's happening is:

  • Airflow is introspecting the operator associated with each task
  • Each task instance is logged into the database table task_instance
    • This table has an attribute called operator
  • When the scheduler re-introspects the code, it looks for the task_instance with the correct operator type; not seeing it, it updates the associated database record(s) as state = 'removed'
  • When the DAG subsequently schedules, airflow

You can see tasks impacted by this process with the query:

SELECT *
FROM task_instance
WHERE state = 'removed'

It looks like there's been work on this issue for airflow 1.10:

That being said, I'm not 100% sure based on the commits that I can find that it would resolve this issue. It seems like the general philosophy is still "when a DAG changes, you should increment / change the DAG name".

I don't love that solution, because it makes it hard to iterate on what is fundamentally one pipeline. The alternative I used was to follow (partially) the recommendations from Astronomer and "blow out" the DAG history. In order to do that, you need to:

  • Stop the scheduler
  • Delete the history from the dag
    • This should result in the DAG completely disappearing from the web UI
    • If it doesn't completely disappear, somewhere the scheduler is still running
  • Restart the scheduler
    • Note: if you're running the DAG on a schedule, be prepared for it to backfill / catchup / run its latest schedule, because you've removed the history
    • If you don't want it to do this, Astronomer's "Fast Forward a DAG" suggestions could be applied
Luis
  • 1,305
  • 5
  • 19
  • 46
Adam Bethke
  • 1,028
  • 2
  • 19
  • 35
0

This can happen when the task status was manually changed (likely through the "Mark Success" option), or forced into a state (as in upstream_failed) and the task never receives a hostname value on the record and wouldn't have any logs or PID

joebeeson
  • 4,159
  • 1
  • 22
  • 29