0

I am using Airflow on an Azure VM. I would like my DAG to run every day at midnight but I cannot always be connected to the VM at this time to run the "airflow scheduler" command. I would like my DAG to be able to run at midnight when the VM is on without my intervention.

I have tried to use the "run command" from the Azure VM manager but the run is limited to 90 minutes. I have also tried to explore the airflow.cfg file but I did not find anything.

Here are the configs of my DAG

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2019, 7, 29),
    'retries': 1,
    'email': ['n.haack@live.fr'],
    'retry_delay': timedelta(minutes=2)}

dag = DAG(dag_id=DAG_ID,
          default_args=default_args,
          schedule_interval=timedelta(days=1))

I wonder if there is an alternative to the "airflow scheduler" command to be able to run DAGs or if there is a way to run a DAG on Airflow only with the server on

EDIT There is a solution which is to run a bootstrap script with "airflow scheduler --daemon". This way, the bash command will run everytime you start your machine and the daemonization will keep it until it is shut down. Thank you @Chengzhi

Neil Haack
  • 3
  • 3
  • 1
    Does you VM get shut down daily and you spin up a new VM? If not, Have you considered using `airflow scheduler --daemon`? https://airflow.apache.org/cli.html#scheduler – Chengzhi Aug 21 '19 at 03:13
  • Thank you for your answer, I didn't consider this option as I am quite a beginner in computer sciences My VM is shut down daily but I don't spin up to a new VM. From what I read, daemonize a command will keep it until the machine is shut down no ? – Neil Haack Aug 21 '19 at 03:31
  • I haven't used Azure VM before, but can you run a bootstrap script to start airflow? Usually, we have revered node to have airflow running 24/7, but it seems you have downtime daily – Chengzhi Aug 21 '19 at 04:15
  • 1
    Ok thank you very much, I think I have figured it out Yes it is possible to run a bootstrap script but independantly from Azure https://stackoverflow.com/questions/12973777/how-to-run-a-shell-script-at-startup – Neil Haack Aug 21 '19 at 07:20

1 Answers1

0

There is a solution which is to run a bootstrap script with "airflow scheduler --daemon". This way, the bash command will run everytime you start your machine and the daemonization will keep it until it is shut down. Thank you @Chengzhi

Elements of response here

Neil Haack
  • 3
  • 3