Meltano with MWAA

Question

I have an AWS MWAA environment to run an ETL Pipeline, and I was planning to consider using Meltano in it. I seem to find no example(s) or documentation that is specific to MWAA. So, I was wondering if that is possible? also, I know that its difficult to SSH into the MWAA Env .. therefore i see it not possible, if its not .. any other recommendations to host Airflow with Meltano? Thanks,

I tried installing Meltano in MWAA (requirements.txt file) but it seems a bit difficult to utilize it in MWAA ..

What issues are you having with Meltano and requirements.txt - does it work or does it fail with any specific errors? — Ricardo Sueiras, Jun 20 '23 at 09:07

score 1 · Accepted Answer · answered Jun 20 '23 at 09:06

I am not familiar with Meltano, but had a quick look at the docs over on GitHub (looks interesting).

It might be possible to run Meltano in Airflow using the BashOperator, after installing the required libraries. From the docs it looks like you might need to run a string of commands, and so you would need to chain these within your BashOperator. Remember, tasks are stateless so you cannot distribute these commands across different tasks (unless Meltano works that way)

The other approach, is to containerise your Meltano stuff and then orchestrate that using one of the container operators (ECSOperator, KubernetesPodOperator, for example)

Meltano with MWAA

1 Answers1