2

I am setting up an AWS MWAA instance.

I have custom operators which themselves reference other python files. I followed the directory structure suggested here (by astronomer.io) and am able to deploy my airflow environment locally without issue.

However, when moving my code base to the S3 bucket, the AWS service isn't able to find my custom operators.

My file structure on the S3 bucket looks like this:

  • s3://{my-bucket-name}
    • dags
      • <dag 1>.py
      • etc...
    • requirements.txt
    • plugins.zip

And my plugins.zip file structure looks like

  • plugins.zip
    • libs
      • <my custom lib 1>.py
      • etc...
    • operators
      • <operator that imports custom lib 1>.py
      • etc...

However I am getting the error:

ModuleNotFoundError: No module named 'operators'

from python once the service starts.

I know some docs use the airflow "plugin" structure for these additional modules, but that seems unnecessary and is even recommended against in the first link I shared here:

...According to the Airflow documentation, [plugins] can be added using Airflow’s Plugins mechanism. This however, overcomplicates the issue and leads to confusion for many people. Airflow is even considering deprecating using the Plugins mechanism for hooks and operators going forward.

Does anyone know what file structure to use for importing simple custom python modules and operators in MWAA?

Maile Cupo
  • 636
  • 1
  • 10
  • 26
  • Are there `__init__.py` files in the directories you want to import? – Josh Fell Aug 31 '21 at 21:07
  • Per https://stackoverflow.com/questions/448271/what-is-init-py-for I didn't think I needed `__init__.py` files so no. – Maile Cupo Aug 31 '21 at 21:39
  • Maybe try this using the mwaa local runner. It might give you better visibility: https://github.com/aws/aws-mwaa-local-runner – Eman Sep 02 '21 at 08:10

2 Answers2

0

Your DAGs will be sync'ed from S3 to /usr/local/airflow/dags/. Your requirements.zip will be unzipped into /usr/local/airflow/plugins/.

So make sure your import statements reflect that folder structure.

dovregubben
  • 364
  • 2
  • 16
0

In my case, the plugins I was trying to integrate had import statements that are no longer supported by v2.02. Import statements that follow "airflow.{operators, sensors, hooks}.<plugin_name> are not supported.

ajjabajja
  • 19
  • 2