Airflow Relative Importing Outside /dag Directory

Question

I haven't been able to move common code outside of the dag directory that airflow uses. I've looked in the airflow source and found imp.load_source.

Is it possible to use imp.load_source to load modules that exist outside of the dag directory? In the example below this would be importing either foo or bar from the common directory.

── airflow_home
     |──── dags
     │   ├── dag_1.py
     │   └── dag_2.py
     ├── common
         ├── foo.py
         └── bar.py

does all these folders part of same app-module? or different modules. just in case all are from same app module you can use relative path to import. — Hara, Jul 13 '17 at 03:03
@Haranadh yes. I updated the diagram to show the airflow_home directory. — cameres, Jul 13 '17 at 05:30
I could not answer in comments. I posted my detailed answer. if you like it. please upvote. :) — Hara, Jul 13 '17 at 06:00
A workaround by modifying the PYTHONPATH is posted here: https://stackoverflow.com/a/48154139/1881080 — cameres, May 29 '18 at 15:08

score 10 · Answer 1 · edited Feb 20 '21 at 14:50

Just add __init__.py files in all 3 folders. it should work. Infact every folder in my folder structure having __init__.py. I could run the code and see output.

Example folder structure could be as:

── airflow_home
     ├── __init__.py
     |──── dags
     │   ├── __init__.py
     │   ├── dag_1.py
     │   └── dag_2.py
     ├── common
         ├── __init__.py
         ├── foo.py
         └── bar.py

and dag_1.py code can be as:

from stackoverflow.src.questions.airflow_home.common.bar import bar_test

def main():
    bar_test()

main()

I'm running this piece of code from my pycharm. Your airflow_home's folder path in my pycharm is stackoverflow/src/questions/airflow_home/

And bar.py code is

def bar_test():
    print "bar hara"

mine airflow home path is /usr/local/airflow... the from statement is like: "from usr.local.airflow" ... It claims an error "no module name usr". What should I do? — Julinho da Adelaide, Jul 17 '18 at 04:44

score 7 · Answer 2 · answered Feb 03 '20 at 10:11

7

Other way instead of adding __init__.py file is to add the following include at the top of dag script:

import sys
import os
sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))

answered Feb 03 '20 at 10:11

Nguyen Van Duc

1,019
10
9

Haresh Chhelana · Answer 3 · 2020-05-16T13:57:46.797

7

Add your airflow home path to PYTHONPATH

export AIRFLOW_HOME=/usr/local/airflow
export PYTHONPATH="${PYTHONPATH}:${AIRFLOW_HOME}"

Dockerfile

ENV AIRFLOW_HOME=/usr/local/airflow
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"

edited May 16 '20 at 13:57

answered Apr 15 '20 at 10:46

Haresh Chhelana

24,720
5
57
67

score 1 · Answer 4 · edited Dec 31 '22 at 03:15

1

This works for me, I'm running airflow on docker by the way:

import sys
import os
sys.path.append(os.path.abspath(os.environ["AIRFLOW_HOME"]))

edited Dec 31 '22 at 03:15

Alexander L. Hayes

3,892
4
13
34

answered Dec 30 '22 at 23:54

andarroyave

11
1

score 0 · Answer 5 · answered May 06 '21 at 00:45

0

After running into the same problem, this fixed it for me:

import sys, os sys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(file),os.path.pardir))) from common.foo import *

answered May 06 '21 at 00:45

Nick Weimer

497
5
7

Airflow Relative Importing Outside /dag Directory

5 Answers5

Linked