Stack Overflow
Stack Overflow

Questions tagged [mwaa]

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed service for Apache Airflow that makes it easy for you to build and manage your workflows in the cloud.

you can easily combine data using any of Apache Airflow’s open source integrations.

use the same familiar Airflow platform as you do today to manage their workflows and now enjoy improved scalability, availability, and security without the operational burden of having to manage the underlying infrastructure

automatically scales capacity up to meet demand and back down to conserve resources and minimize costs

integrated with AWS security services to enable secure access

manages the provisioning and ongoing maintenance of Apache Airflow

https://docs.aws.amazon.com/mwaa/

291 questions
9
votes
3 answers

Setting custom environment variables in managed apache airflow

We are planning to switch from managing airflow ourselves to Managed Apache Airflow services of AWS. Our original dags use some custom environment variables that need to be set in Managed airflow as well. So far I was not able to find a way to set…
Vibhav
  • 181
  • 5
  • 11
8
votes
4 answers

How to use DBT with AWS Managed Airflow?

hope you are doing well. I wanted to check if anyone has get up and running with dbt in aws mwaa airflow. I have tried without success this one and this python packages but fails for some reason or another (can't find the dbt path, etc). Did anyone…
nariver1
  • 353
  • 3
  • 19
7
votes
1 answer

MWAA Airflow 2.2.2 'DAG' object has no attribute 'update_relative'

So I was upgrading DAGs from airflow version 1.12.15 to 2.2.2 and DOWNGRADING python from 3.8 to 3.7 (since MWAA doesn't support python 3.8). The DAG is working fine on the previous setup but shows this error on the MWAA setup: Broken DAG:…
Salahddin-Warid
  • 105
  • 1
  • 1
  • 5
7
votes
1 answer

MWAA in productions - tasks queued for unknown reasons

Does anyone use MWAA in production? We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unknown reasons. Task is in the 'queued' state which is not a valid state for execution. The…
val
  • 329
  • 2
  • 16
7
votes
3 answers

Is it possible to access the Airflow API in AWS MWAA?

I have an AWS MWAA Airflow v2.0.2 instance running. I would like to have access to the Airflow API. Is this something supported currently? Planned for future releases? There is no mention of this in the AWS docs. It looks like it was possible to…
ypicard
  • 3,593
  • 3
  • 20
  • 34
7
votes
3 answers

MWAA Airflow 2.0 in AWS Snowflake connection not showing

Snowflake is not showing in the connections dropdown. I am using MWAA 2.0 and the providers are already in the requirements.txt MWAA uses python 3.7 dont know if this can be a thing Requirements.txt: --constraint…
AXI
  • 141
  • 1
  • 5
6
votes
0 answers

Running docker container as amd64 machine using a M1 Mac

I have a Docker file which creates an image and then I run it using docker compose together with a container built using a Postgres image. (To set up a local environment of Airflow - we use the mwaa local runner). Recently I got a new M1 pro machine…
snir.isl
  • 155
  • 1
  • 12
6
votes
2 answers

MWAA - Airflow - PythonVirtualenvOperator requires virtualenv

I am using AWS's MWAA service (2.2.2) to run a variety of DAGs, most of which are implemented with standard PythonOperator types. I bundle the DAGs into an S3 bucket alongside any shared requirements, then point MWAA to the relevant objects &…
trilson86
  • 939
  • 1
  • 9
  • 20
6
votes
1 answer

Can MWAA requirements.txt be automatically set to the latest version?

We have an automated upload system for our DAG's to MWAA. Is there a way to have the requirements.txt in the S3 bucket be automatically set to its latest version? I cannot find an option for this in the AWS console.
SoftwareThings
  • 304
  • 4
  • 15
6
votes
2 answers

No module named 'airflow.providers.ssh' on AWS Airflow (Amazon MWAA)

I need to use sshoperator in a DAG on AWS Airflow (Amazon MWAA), so I imported the following library in my DAG file from airflow.contrib.operators.ssh_operator import SSHOperator It seems sshoperator has been defined in paramiko library, so I have…
Sam
  • 426
  • 6
  • 16
5
votes
2 answers

AWS Managed Airflow upload a folder to MWAA environment

I'm going to start using AWS-managed airflow. For the managed airflow to access the dags, I need to upload my code to the dags/ directory in an s3 bucket, and MWAA will pick it up. However, in my codebase, I have codes in other directories, for…
taraf
  • 777
  • 2
  • 10
  • 28
5
votes
4 answers

AWS managed Ariflow doesn't fetch log

I am using an AWS Managed Airflow to orchestrate a data pipeline. The problem that I am facing now is that logs are not being printed in the airflow log not on cloud watch. I am getting the following error in the log of Air flow: *** Reading remote…
Birat Bade Shrestha
  • 800
  • 1
  • 8
  • 28
5
votes
2 answers

Amazon MWAA Airflow - Tasks container shut down / stop / killed without logs

We use Amazon MWAA Airflow, rarely some task as marked as "FAILED" but there is no logs at all. As if the container had been shut down without noticing us. I have found this…
tdebroc
  • 1,436
  • 13
  • 28
4
votes
1 answer

MWAA Airflow Scaling: what do I do when I have to run frequent & time consuming scripts? (Negsignal.SIGKILL)

I have an MWAA Airflow env in my AWS account. The DAG I am setting up is supposed to read massive data from S3 bucket A, filter what I want and dump the filtered results to S3 bucket B. It needs to read every minute since the data is coming in every…
Kei
  • 611
  • 2
  • 11
  • 24
4
votes
1 answer

How to force delete an MWAA environment?

I have updated some Airflow configuration options in my MWAA environment and it has been in "Updating" state for the last three hours! Does anybody know how to force-delete that environment? Using the "Delete" button on the AWS MWAA console only…
Babak Tourani
  • 419
  • 2
  • 6
  • 18
1
2 3
…
19 20