0

I have an airflow task where I try and load a file into an s3 bucket. I have airflow running on a Ec2 instance. Im running AF version 2.4.3 I have done

pip install 'apache-airflow[amazon]'

I start up my AF server, log in and go to the Admin section to add a connection. I open a new connection and I dont have an option for s3.

My only Amazon options are: Amazon Elastic MapReduce Amazon Redshift Amazon Web services.

what else am I missing?

aero8991
  • 239
  • 1
  • 13

3 Answers3

1

You need to define aws connection under "Amazon Web Services Connection"

for more details see here

ozs
  • 3,051
  • 1
  • 10
  • 19
0

You should define the connection within your DAG.

You should also use a secure settings.ini file to save your secrets, and then call those variables from your DAG.

See this answer for a complete guide: Airflow s3 connection using UI

chipsnbits
  • 1
0

I am using Airflow version 2.4.3

I also did:

pip install 'apache-airflow[amazon]'

and also only had the options of the following in the UI:

  • Amazon Elastic MapReduce
  • Amazon Redshift
  • Amazon Web services

Other tutorials (maybe older versions?) show that there should be Amazon S3 as an option.

However, if you select Amazon Web Services, then add you configuration in the "Extra" field:

{"aws_access_key_id": "<insert here>", "aws_secret_access_key": "<insert here>"}

Then it will work.

PoppyB
  • 1