AirFlow SIGTERM While EMR Job is Running

Asked Mar 17 '22 at 19:05

Active Mar 17 '22 at 19:05

Viewed 358 times

I have an AriFlow DAG where each step is an AWS EMR task. Once AirFlow reaches one of the steps, it sends the SIGTERM signal as the following

{emr_step.py:73} INFO - Poking step XXXXXXXX on cluster XXXXXXXX
{emr_base.py:66} INFO - Job flow currently RUNNING
{local_task_job.py:199} WARNING - State of this instance has been externally set to failed. Terminating instance.
{process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 7632

This is in spite of the fact that the EMR job is still running healthy. One major difference between the EMR job that AirFlow fails on and the rest of my EMR jobs is that it triggers anther system and wait to hear back from that system. In other words, it stays idle until it hears back from aother system. My impression is that AirFlow thinks the EMR job has failed. However, it is just waiting to hear from another system.

Is there any way to ask AirFlow to wait more for this EMR job?

asked Mar 17 '22 at 19:05

A.M.

1,757
5
22
41

Try this answer out https://stackoverflow.com/a/70128529/14091289 – caxefaizan Mar 25 '22 at 17:33
I had tried that but it didn't help. – A.M. Mar 29 '22 at 23:41
Every operator has execution_timeout (datetime.timedelta) – max time allowed for the execution of this task instance, if it goes beyond it will raise and fail. Have you tried setting this? – caxefaizan Mar 31 '22 at 00:48

AirFlow SIGTERM While EMR Job is Running

0 Answers0