MWAA in productions - tasks queued for unknown reasons

Question

Does anyone use MWAA in production?

We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unknown reasons.

Task is in the 'queued' state which is not a valid state for execution. The task must be cleared in order to be run.

It happens randomly, can perfectly run for a day and then a few tasks will stay queued. The tasks will stay in this state forever unless we mark them as failed manually.

A DAG run can stay in this "queued" state even if the pool is empty, I don't see any reasons explaining this.

It happens to ~5% of the tasks with all the others running smoothly.

Did you ever encounter this behavior?

whats the version of airflow you are using? – Priyadarshan Mohanty May 31 '22 at 17:28 — Priyadarshan Mohanty, May 31 '22 at 17:28
I'm seeing the same thing occurring for `MWAA v.2.2.2` – nate Sep 02 '22 at 18:17 — nate, Sep 02 '22 at 18:17

Kevin Vo · Accepted Answer · 2022-06-17T17:08:43.377

6

This was happening to me in MWAA as well. The solution, recommended to me by AWS, was adding to Airflow configuration options via the web UI the following options:

celery.sync_parallelism = 1 
core.dag_file_processor_timeout = 150
core.dagbag_import_timeout = 90 
core.min_serialized_dag_update_interval = 300
scheduler.dag_dir_list_interval = 600 
scheduler.min_file_process_interval = 300
scheduler.parsing_processes = 2 
scheduler.processor_poll_interval = 60

edited Jun 17 '22 at 17:08

answered Apr 20 '22 at 20:15

Kevin Vo

191
1
4

1

Did they say why this would work? What was the issue? – eljusticiero67 Jan 16 '23 at 19:14

MWAA in productions - tasks queued for unknown reasons

1 Answers1