6

We have an automated upload system for our DAG's to MWAA.

Is there a way to have the requirements.txt in the S3 bucket be automatically set to its latest version?

I cannot find an option for this in the AWS console.

SoftwareThings
  • 304
  • 4
  • 15

1 Answers1

4

It's in the setup of MWAA. Edit MWAA instance, in the requirements.txt field, set the version to the latest.

Also,

MWAA uses requirements file to create the container image. So when you upload requirement file and specify it in the edit options and save, images are created(it takes couple of minutes to do it). Ideally image should only be updated when you have new python libraries to add. if you still want to automate, try to use lambda S3 trigger and use mwaa cli to trigger the update-environment command.

joss
  • 695
  • 1
  • 5
  • 16
  • 1
    It does allow me to select the most recent version in the bucket, but the question is asking if there is a method to automatically set the `requirements.txt` to the latest version – SoftwareThings Nov 18 '21 at 14:12
  • What do you mean automatically, When you upload a new file, it is automatically tagged as the latest version. – joss Nov 18 '21 at 16:18
  • 1
    When I upload a new `requirements.txt`, I want that to be set as the `requirements.txt` used by MWAA without me having to go into the AWS console and set it myself manually – SoftwareThings Nov 18 '21 at 16:47
  • 1
    MWAA uses requirement file to create the container image. So when you upload requirement file and specify it in the edit options and save, images are created(it takes couple of minutes to do it). Ideally image should only be updated when you have new python libraries to add. if you still want to automate, try to use lambda S3 trigger and use mwaa cli to trigger the update-environment command. – joss Nov 18 '21 at 20:41
  • Thanks! If you update your answer to include your previous comment, I will mark it accepted :) – SoftwareThings Nov 19 '21 at 13:28