Airflow is unable to Import custom python package

Question

I want to call a script through airflow from a custom python project

My directory structure is:

/home/user/
      ├──airflow/
      │  ├──dags
             ├──.venv_airflow (virtual environment for airflow)
      │      ├──my_dag.py
      ├──my_project
         ├──.venv (virtual environment for my_project)
         ├──folderA
            ├──__init__.py
            ├──folderB
               ├──call_me.py (has a line "from my_project.folderA.folderB import import_me")
               ├──import_me.py

My dag file looks like:

from airflow import DAG
import datetime as dt
from airflow.operators.bash_operator import BashOperator

default_args = {
    'owner': 'arpita',
    'start_date': dt.datetime(2019, 11, 20),
    'retries': 1,
    'retry_delay': dt.timedelta(minutes=5),
    'depends_on_past': False,
    'email': ['example@abc.com'],
    'email_on_failure': True,
    'email_on_retry': True,
}

with DAG('sample',
         default_args=default_args,
         schedule_interval='30 * * * *',
         ) as dag:
    enter_project = BashOperator(task_id='enter_project',
                                 bash_command='cd /home/user/my_project',
                                 retries=2)
    setup_environment = BashOperator(task_id='setup_environment',
                                     bash_command='source /home/user/my_project/.venv/bin/activate',
                                     retries=2)
    call_script = BashOperator(task_id='call_script',
                                 bash_command='python -m my_project.folderA.folderB.call_me,
                                 retries=2)

enter_project >> setup_environment >> call_script

But I am getting this error

[2019-11-22 11:56:49,311] {bash_operator.py:115} INFO - Running command: python -m my_project.folderA.folderB.call_me
[2019-11-22 11:56:49,315] {bash_operator.py:124} INFO - Output:
[2019-11-22 11:56:49,349] {bash_operator.py:128} INFO - /home/user/airflow/.venv/bin/python: Error while finding spec for 'my_project.folderA.folderB.call_me' (ImportError: No module named 'my_project')

Project and the script are working outside airflow. In airflow, it imports other packages like pandas and tensorflow but not custom packages. I tried inserting path with sys.path.insert but that is not working. Thank you for reading:)

Weird, according to https://stackoverflow.com/questions/56134683/how-to-import-external-scripts-in-a-airflow-dag-with-python sys.path should have worked — Dolfa, Nov 22 '19 at 13:00

score 0 · Accepted Answer · answered Nov 22 '19 at 18:54

Your bash commands run in three separate bash operators. It should run in one.

call_script = BashOperator(
    task_id='call_script',
    bash_command='cd /home/user/my_project;'
                 'source /home/user/my_project/.venv/bin/activate;'
                 'python -m my_project.folderA.folderB.call_me',
    retries=2)

Airflow is unable to Import custom python package

1 Answers1