How to specify credentials when connecting to boto3 S3?

Question

On boto I used to specify my credentials when connecting to S3 in such a way:

import boto
from boto.s3.connection import Key, S3Connection
S3 = S3Connection( settings.AWS_SERVER_PUBLIC_KEY, settings.AWS_SERVER_SECRET_KEY )

I could then use S3 to perform my operations (in my case deleting an object from a bucket).

With boto3 all the examples I found are such:

import boto3
S3 = boto3.resource( 's3' )
S3.Object( bucket_name, key_name ).delete()

I couldn't specify my credentials and thus all attempts fail with InvalidAccessKeyId error.

How can I specify credentials with boto3?

This answer might help you: https://stackoverflow.com/a/36913771/2681632 — Ilja Everilä, Aug 31 '17 at 13:00
See the "Configuring Credentials" section in the official documentation: https://boto3.readthedocs.io/en/latest/guide/configuration.html — Mark B, Aug 31 '17 at 13:14
I find it super strange to call this 'AWS_SERVER_PUBLIC_KEY'. The name is 'access key id' and has nothing to do with the public part of a keypair. Also an access to a service like s3 should not be confused with a server(host) access. And i recommend to not let this key id becoming public (even if it's useless alone) — mathieu, Jan 21 '22 at 17:28

score 326 · Accepted Answer · answered Aug 31 '17 at 13:00

326

You can create a session:

import boto3
session = boto3.Session(
    aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
    aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
)

Then use that session to get an S3 resource:

s3 = session.resource('s3')

answered Aug 31 '17 at 13:00

Alasdair

298,606
55
578
516

65

works, I will take it as the answer. Why on earth don't they document this as the obvious way to do it?!! – Robert Brax Aug 31 '17 at 13:07
5

As mentioned in a comment above, this _is_ in fact in the [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html). – Moot Oct 20 '18 at 08:06
1

@Moot I was initially going to say I couldn't find this in the docs but under [credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) in the 'warning' section it notes that the token is optional. I don't really get why you would ever bother with that. – JimmyJames Feb 03 '21 at 16:21
1

@JimmyJames this is getting off topic, but you can use AWS STS to generate temporary credentials (e.g. valid for one hour). In that case, the session token is required, it won't work if you omit it. – Alasdair Feb 03 '21 at 17:51
@Alasdair Thanks, I appreciate the response. Perhaps I'll create a new question because this still doesn't make sense to me. If you could pass just a token, then sure but it seems the secret key is always required and it's the same key that can be used to get a token. – JimmyJames Feb 03 '21 at 18:30
3

@JimmyJames the use case for STS is that you start with `aws_access_key_id` and `aws_secret_access_key` which have limited permissions. They don't allow you access S3, but they do allow you to [assume a role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) which can access S3. You make the AWS STS call to assume the role, which returns an new `aws_access_key_id`, `aws_secret_access_key` and `aws_session_token` combination (the key and access key are different from the originals). You then use these credentials to create a new session to access S3. – Alasdair Feb 03 '21 at 20:13
@Alasdair Thanks. That the keys were also generated as part of that was what I was missing. I don't see why a token alone couldn't be made to be good enough (e.g. concat the other keys to it) but that's not a practical issue. – JimmyJames Feb 03 '21 at 20:42

score 160 · Answer 2 · edited May 15 '18 at 10:51

160

You can get a client with new session directly like below.

 s3_client = boto3.client('s3', 
                      aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, 
                      aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY, 
                      region_name=REGION_NAME
                      )

edited May 15 '18 at 10:51

Alasdair

298,606
55
578
516

answered Aug 31 '17 at 13:00

Rajez

3,717
1
14
21

13

This works for getting an s3 *client*, but the OP wanted an s3 *resource* instead. – Alasdair May 15 '18 at 10:52
2

I agree with @Alasdair. The docs don't show how to do anything with client, and neither do you, so I don't see how this answer is relevant. – Cerin May 14 '19 at 19:51
I tried this but it gives me "Unable to locate credentials" error..I had previously removed ~/.aws folder to test this as i know that boto will by default look for creds there... – Prathamesh dhanawade Dec 03 '19 at 20:54
This is the right answer and the only method that works as today. I don't know what you guys are talking about this not being useful. You can do ANYTHING using the client and there's extensive documentation for EVERY AWS service. Just take a look for S3: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html – Dan Dec 07 '21 at 05:08
@Dan whilst Client is very useful, there are additional constraints over Resource. For instance, Client can only receive info for 1000 objects at a time. – LotiLotiLoti Jul 25 '23 at 13:34

score 18 · Answer 3 · answered Jan 15 '20 at 18:30

This is older but placing this here for my reference too. boto3.resource is just implementing the default Session, you can pass through boto3.resource session details.

Help on function resource in module boto3:

resource(*args, **kwargs)
    Create a resource service client by name using the default session.

    See :py:meth:`boto3.session.Session.resource`.

https://github.com/boto/boto3/blob/86392b5ca26da57ce6a776365a52d3cab8487d60/boto3/session.py#L265

you can see that it just takes the same arguments as Boto3.Session

import boto3
S3 = boto3.resource('s3', region_name='us-west-2', aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY, aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY)
S3.Object( bucket_name, key_name ).delete()

score 14 · Answer 4 · answered Oct 08 '19 at 16:18

I'd like expand on @JustAGuy's answer. The method I prefer is to use AWS CLI to create a config file. The reason is, with the config file, the CLI or the SDK will automatically look for credentials in the ~/.aws folder. And the good thing is that AWS CLI is written in python.

You can get cli from pypi if you don't have it already. Here are the steps to get cli set up from terminal

$> pip install awscli  #can add user flag 
$> aws configure
AWS Access Key ID [****************ABCD]:[enter your key here]
AWS Secret Access Key [****************xyz]:[enter your secret key here]
Default region name [us-west-2]:[enter your region here]
Default output format [None]:

After this you can access boto and any of the api without having to specify keys (unless you want to use a different credentials).

You can also specify the column you want to fill : - `aws configure set aws_access_key_id [****************ABCD]` - `aws configure set aws_secret_access_key [****************xyz]` — mjeday, Jan 10 '22 at 14:49

score 9 · Answer 5 · answered Sep 12 '21 at 12:13

If you rely on your .aws/credentials to store id and key for a user, it will be picked up automatically.

For instance

session = boto3.Session(profile_name='dev')
s3 = session.resource('s3')

This will pick up the dev profile (user) if your credentials file contains the following:

[dev]
aws_access_key_id = AAABBBCCCDDDEEEFFFGG
aws_secret_access_key = FooFooFoo
region=op-southeast-2

score 2 · Answer 6 · answered Nov 08 '18 at 23:41

There are numerous ways to store credentials while still using boto3.resource(). I'm using the AWS CLI method myself. It works perfectly.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html?fbclid=IwAR2LlrS4O2gYH6xAF4QDVIH2Q2tzfF_VZ6loM3XfXsPAOR4qA-pX_qAILys

score 0 · Answer 7 · answered Jun 08 '21 at 20:35

0

you can set default aws env variables for secret and access keys - that way you dont need to change default client creation code - though it is better to pass it as a parameter if you have non-default creds

answered Jun 08 '21 at 20:35

Den Roman

548
7
10

How to specify credentials when connecting to boto3 S3?

7 Answers7

Linked

Related