How to import external scripts in a Airflow DAG with Python?

Question

I have the following structure:

And I try to import the script inside some files of the inbound_layer like so:

import calc

However I get the following error message on Airflow web:

Any idea?

if you are using pycharm and add the package in project dependency than you can import it so , else you have to add the desired package in path to run it. — sahasrara62, May 14 '19 at 16:24
Sounds like to need to add the package to your path. See [How to import a Python class that is in a directory above?](https://stackoverflow.com/a/11096846/8150685) as an example. — Error - Syntactical Remorse, May 14 '19 at 16:28

score 4 · Answer 1 · edited May 30 '19 at 03:51

4

For airflow DAG, when you import your own module, you need make sure 2 things:

where is the module? You need to find where is the root path in you airflow folder. For example, in my dev box, the folders are:

~/projects/data/airflow/teams/team_name/projects/default/dags/dag_names/dag_files.py

The root is airflow, so if I put my modules my_module in

~/projects/data/airflow/teams/team_name/common

Then I need to use

from teams.team_name.common import my_module

In your case, if the root is the upper folder of bi, and you put the scripts of calc in bi/inbound_layer/test.py then you can use:

from bi.inbound_layer.test import calc

And you must make sure you have \__init\__.py files in the directory structure for the imports to function properly. You should have an empty file \__init\__.py in each folder in the path. It indicates this directory is part of airflow packages. In your case, you can use touch \__init\__.py (cli) under bi and _inbound_layer_ folders to create the empty __init\__.py.

edited May 30 '19 at 03:51

Felipe Augusto

7,733
10
39
73

answered May 14 '19 at 17:12

AC at CA

695
6
13

What do you mean about the root path? is that the path where my dags are loaded? – Felipe Augusto May 14 '19 at 17:45
Here, the path root I mentioned is the path defined in $PYTHONPATH. You can use "echo $PYTHONPATH" in you env to get it. BTW please vote on the answer. – AC at CA May 14 '19 at 18:14
Or, you can simply just add "bi" into your sys path. – AC at CA May 14 '19 at 18:41
Thank you, I'm gonna post my specific solution, you tips really helped me! – Felipe Augusto May 14 '19 at 19:23

score 4 · Answer 2 · answered Apr 01 '21 at 15:27

4

Airflow adds dags/, plugins/, and config/ directories in the Airflow home to PYTHONPATH by default so you can for example create folder commons under dags folder, create file there (scriptFileName ). Assuming that script has some class (GetJobDoneClass) you want to import in your DAG you can do it like this:

from common.scriptFileName import GetJobDoneClass

answered Apr 01 '21 at 15:27

Hrvoje

13,566
7
90
104

2

That's work. What happen if we don't want to have it under dags? and just another folder next to dags. How does it work? – pm1359 May 18 '21 at 18:01

Felipe Augusto · Accepted Answer · 2019-05-14T19:33:47.897

1

I needed insert the following script inside at the top of ren.py :

import sys, os
from airflow.models import Variable

DAGBAGS_DIR = Variable.get('DAGBAGS_DIR')
sys.path.append(DAGBAGS_DIR + '/bi/inbound_layer/')

This way I make available the current folder packages.

edited May 14 '19 at 19:33

answered May 14 '19 at 19:25

Felipe Augusto

7,733
10
39
73

How to import external scripts in a Airflow DAG with Python?

3 Answers3

Linked