1

Let me know if any of these got improvements.

  1. Airflow 2.0.2 and lower constantly faces issues with respect to the mini scheduler.
[2022-02-21 08:40:23,027] {{process_utils.py:100}} INFO - Sending Signals.SIGTERM to GPID 4081
[2022-02-21 08:41:23,036] {{process_utils.py:113}} WARNING - process psutil.Process(pid=4081, name='airflow task ru', status='sleeping', started='08:40:17') did not respond to SIGTERM. Trying SIGKILL
[2022-02-21 08:41:23,044] {{process_utils.py:66}} INFO - Process psutil.Process(pid=4081, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='08:40:17') (4081) terminated with exit code Negsignal.SIGKILL
[2022-02-21 08:41:23,044] {{standard_task_runner.py:130}} ERROR - Job 9926 was killed before it finished (likely due to running out of memory)

And the recommendation is to change schedule_after_task_execution to False if True which is the default.

Task supervisor process runs a "mini scheduler". However, after marking it as success, it will call _run_mini_scheduler_on_child_tasks. And while local_task_job.py will detect the task as success and not running, it will terminate the process which might be still executing under _run_mini_scheduler_on_child_tasks.

Sometimes this is undesirable.

  1. Airflow scheduler constantly parses the python file for building and running the tasks. Any code in DAG file but that is outside the task runs multiple times even though the DAG is scheduled to run just once a day. Any improvements to this?
John Eipe
  • 10,922
  • 24
  • 72
  • 114

1 Answers1

0

  1. There is no reason to change this setting unless you experience problems. Example for when you might consider changing it is in https://stackoverflow.com/a/71196200/14624409

  2. No. Airflow must scan the DAG folder searching for new DAGs so any top level code that you have in your dag .py files will be parsed every min_file_process_interval which defaults to 30 seconds. There will be future improvements on somehow related area in 2.3.0 when AIP-42 Dynamic Task Mapping will be completed and allow map-reduce like design

Elad Kalif
  • 14,110
  • 2
  • 17
  • 49
  • Thanks. For issue (1): I updated the question with logs. For issue (2): I remember in one of the talks that it is being addressed. Never new at what stage it is in. – John Eipe Feb 24 '22 at 17:09
  • Sigterm means you are out of memory for some reason... I dont think its related. – Elad Kalif Feb 24 '22 at 17:32
  • And after changing it to False this intermittent issue stopped happening – John Eipe Feb 24 '22 at 20:26
  • i'd suggest to keep monitoring it. Whether you turn it off or on - no exception should be raised. This is just effecting performance but should not cause scheduler to crash. If you can isolate the cause and have a reproduce example please open Github issue to Airflow – Elad Kalif Feb 24 '22 at 20:35