I am trying to package my Repository with my Dag in a Zip file like it states here in the documentation. So i have followed the convention in the documentation, which is to keep the dag in the root of the zip, and the sub directories are viewed as packages by airflow.
My zip file has the following contents:
$ unzip -l $AIRFLOW_HOME/dags/test_with_zip.zip
Archive: /home/arjunc/Tutorials/airflow/dags/test_with_zip.zip
Length Date Time Name
--------- ---------- ----- ----
0 2018-03-29 17:46 helloworld/
189 2018-03-29 17:22 helloworld/hello.py
0 2018-03-29 17:18 helloworld/__init__.py
461 2018-03-29 17:24 test_with_zip_dag.py
--------- -------
650 4 files
Where test_with_zip_dag.py is the file in the root directory with the Dag definitions as follows:
from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from helloworld.hello import HelloWorld
def run():
return HelloWorld().run()
dag = DAG('test_with_zip', description='Test Dependencies With Zipping',
schedule_interval='0 12 * * *',
start_date=datetime(2017, 3, 20), catchup=False)
hello_operator = PythonOperator(task_id='hello_task', python_callable=run, dag=dag)
I have placed this zip in the default dags directory $AIRFLOW_HOME/dags, but my dag isn't recognized!
What am I doing wrong?
Update
When I restarted the webserver, the task test_with_zip has popped up, but it is not runnable because the Scheduler doesn't seem to recognize it. I get the following error for it (from the web interface):
This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence.
