3

I'm trying to use local module inside a dag on MWAA.

The folder structure looks like :

.
├── __init__.py
├── dags
│   ├── __init__.py
│   └── my_dag
│      ├── __init__.py
│      └── dag.py
│   └── utils
│      ├── __init__.py
│      └── file.py
│      └── secrets.py
│      └── date.py

I try to use functions from ./dags/utils/secrets by importing them like :

from dags.utils.secrets import get_secret

Locally, I've been able to make it works by setting environment variable PYTHONPATH to usr/local/airflow

Is it the best way ? If not how can I make it works on MWAA ?

Thank you,

Sébastien
  • 33
  • 4

1 Answers1

3

Locally, I've been able to make it works by setting environment variable PYTHONPATH to usr/local/airflow

Is it the best way ? If not how can I make it works on MWAA ?

When deploying Airflow to an MWAA environment, you don't explicitly set the PYTHONPATH environment variable.

I try to use functions from ./dags/utils/secrets by importing them like :

from dags.utils.secrets import get_secret

Adjust the Python import statement relative to the MWAA environment's DAGs folder. For example, if the DAGs folder is s3://<bucket>/dags, then the import statement would be:

from utils.secrets import get_secret

Example DAGs folder:

s3://<bucket>/dags/__init__.py
s3://<bucket>/dags/my_dag/__init__.py
s3://<bucket>/dags/my_dag/dag.py
s3://<bucket>/dags/utils/__init__.py
s3://<bucket>/dags/utils/file.py
s3://<bucket>/dags/utils/secrets.py
s3://<bucket>/dags/utils/date.py
Andrew Nguonly
  • 2,258
  • 1
  • 17
  • 23
  • The adjustment seems to work perfectly, thank you ! – Sébastien Mar 09 '22 at 10:58
  • No problem @Sébastien! Feel free to accept the answer if the solution is correct. – Andrew Nguonly Mar 09 '22 at 15:16
  • This adjustment obviously works, but it goes against good practices not having the full path in your package imports (and IDEs will complain). Any other idea around? – alete Nov 28 '22 at 20:59
  • I have the same issue, and I have some utils module outside the dags folder from which I make imports, but it doesn't work on MWAA. how can I adjust the absolute path on the cloud please? – Th3FreeSpirit Dec 22 '22 at 11:50
  • There's an issue with this solution. It can't resort imports from the same level. E.g. if there is two sub-modules: `dag/utility/utils.py` and `dag/includes/my_module.py`, the `from utility.utils import *` won't work. MWAA either throws a `ModuleNotFoundError: No module named 'utility'` exception or `ModuleNotFoundError: No module named 'dags'` dependendent on relative or absolute imports used. IMHO MWAA is the worst Airflow service out there. – Gergely M Jun 07 '23 at 11:23