7

I have an AWS MWAA Airflow v2.0.2 instance running.

I would like to have access to the Airflow API. Is this something supported currently? Planned for future releases? There is no mention of this in the AWS docs.

It looks like it was possible to enable the experimental API in AWS MWAA Airflow v1.10.12, but when I try to add api.auth_backend = airflow.api.auth.backend.default in the AWS UI, i get this error message:

Some of the provided configurations belong to the blocklist and can not be applied: api.auth_backend
ypicard
  • 3,593
  • 3
  • 20
  • 34
  • Seems to be asked in https://stackoverflow.com/questions/66344988/how-to-enable-the-api-in-aws-managed-workflows-for-apache-airflow – Tomasz Urbaszek Jun 08 '21 at 10:55
  • Yes, but this gives access to the Airflow CLI, not the Airflow API. Quite a big difference in features actually! – ypicard Jun 09 '21 at 12:17
  • There are two options: a) MWAA does not support Airflow API b) try using `auth_backend = airflow.api.auth.backend.basic_auth` because the default option can be `deny_all` as described in https://airflow.apache.org/docs/apache-airflow/stable/security/api.html (I don't remember when we changed the default) Also - consider migrating to 2.0 as 1.10.X version will reach EOL on 17th of June this year. – Tomasz Urbaszek Jun 09 '21 at 13:25
  • 4
    I am already running v2.0.2. And as mentioned in the post, this `api.auth_backend` is blocklisted by AWS. – ypicard Jun 09 '21 at 13:43

3 Answers3

8

According to THIS SESSION of Airflow Summit 2021 (around 41:15, when attendee ask about API of Airflow 2.0 and security concern). It seems AWS block REST API for security reason at the moment.

Martin Chen
  • 105
  • 11
2

We were able to run some API calls on MWAA CLI as described in the official AWS MWAA User Guide.

Unfortunately, not all Airflow API commands are supported by the MWAA CLI, but the documentation is quite clear about that.

dovregubben
  • 364
  • 2
  • 16
1

Here is how to run a DAG on an AWS MWAA instance using a Node.js Lambda function without changing the api.auth_backend setting:

const axios = require('axios');
const { MWAAClient, CreateCliTokenCommand } = require('@aws-sdk/client-mwaa');

const client = new MWAAClient({ region: 'us-east-2' });

const getAirflowCliToken = async (environmentName) => {
  const command = new CreateCliTokenCommand({ Name: environmentName });
  const { CliToken, WebServerHostname } = await client.send(command);

  return { token: CliToken, host: WebServerHostname };
};

const triggerDag = async (environmentName, dagName, payload) => {
  // Get an Airflow CLI token.
  const { token, host } = await getAirflowCliToken(environmentName);

  const url = `https://${host}/aws_mwaa/cli`;
  const formattedPayload = JSON.stringify(payload);
  const data = `dags trigger -c '${formattedPayload}' ${dagName}`;
  const options = {
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'text/plain'
    }
  };

  await axios.post(url, data, options);
};

const handler = async (event) => {
  const environmentName = 'example-environment'
  const dagName = 'example_dag';
  const payload = {
    task_payload: 'FOOBAR'
  };

  // Trigger Airflow DAG.
  await triggerDag(environmentName, dagName, payload);

  return {
    statusCode: 200,
    body: 'OK'
  };
};

module.exports.handler = handler;
Chad Johnson
  • 21,215
  • 34
  • 109
  • 207
  • Chad Jhonson I did exactly the same as this request, but I keep getting 400 Airflow command parsing error response, is this something has to be configured in the dags ? although using python script actually is successfully running the dag using the same format of the command – Basil Satti Jan 20 '23 at 16:15
  • since the content type is text/plain, I managed to get this worked by axios.post(url, "", options) – Basil Satti Jan 20 '23 at 16:41