3

I have Airflow running in AWS MWAA, I would like to access REST API and there are 2 ways to do this but doesn't seem to work for me.

  1. Overriding api.auth_backend. This used to work and now AWS MWAA won't allow you to add this, it is consider as 'blocklist' and not allow.
    api.auth_backend = airflow.api.auth.backend.default

  2. Using MWAA Cli(Python). This doesn't work if any of the DAGs uses packages that are in requirments.txt file.
    a. as an example, I have "paramiko" in requirements.txt because I have a task that uses SSHOperator. The MWAA Cli fails with "no module paramiko"

    b. Also noted here, https://docs.aws.amazon.com/mwaa/latest/userguide/access-airflow-ui.html "Any command that parses a DAG (such as list_dags, backfill) will fail if the DAG uses plugins that depend on packages that are installed through requirements.txt."

user15364819
  • 61
  • 5
  • Did you manage to find a solution to your problem? – ypicard Jun 07 '21 at 08:51
  • @ypicard Unfortunately no, they did however released Airflow 2.x as part of MWAA. I'm going to test that version. – user15364819 Jun 08 '21 at 13:20
  • I am running the Airflow v2.x in MWAA and the API is not available at the moment in this version. https://stackoverflow.com/questions/67884770/is-it-possible-to-access-the-airflow-api-in-aws-mwaa?noredirect=1#comment119989884_67884770 – ypicard Jun 09 '21 at 12:18
  • @ypicard That's unfortunate. – user15364819 Jun 10 '21 at 13:02

1 Answers1

2

We are using MWAA 2.0.2 and managed to use Airflow's Rest-API through MWAA CLI, basically following the instructions and sample codes of the Apache Airflow CLI command reference. You'll notice that not all Rest-API calls are supported, but many of them are (even when you have a requirements.txt in place).

Also have a look at AWS sample codes on GitHub.

dovregubben
  • 364
  • 2
  • 16