Boto3 error: The AWS Access Key Id you provided does not exist in our records

Question

I am currently trying to get access to Amazon S3 inside a virtual machine and download files like so:

s3 = boto3.resource('s3',
         aws_access_key_id="xxxxxxxxxxx",
         aws_secret_access_key="xxxxxxxxxxxxxxxxx")
s3client = boto3.client('s3')

bucket = s3.Bucket('bucketone')

for obj in bucket.objects.all():
    s3client.download_file(bucket_name, obj.key, filename)

But I’m getting the error:

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

What could I be doing wrong? I checked my aws_access_key_id and aws_secret_access_key multiple times, but still getting the same error. The same code locally, but not on a virtual machine, actually works on a different computer as well. There is a reason why I’m hardcoding in the keys, as I have to.

Answer 1

You need to set the access for the boto3 session. You don't really want to put your keys in your code. What I would recommend doing first is running 'aws configure' and setting your aws_access_key_id and aws_secret_access_key in your .credentials file. Then in your code do the following:

session = boto3.Session(profile_name='name_of_your_profile')

If you have just the default profile, you might not need to do that or for good measure, just put:

session = boto3.Session(profile_name='default')

Once you have that in your code you can establish a connection to s3 with:

s3 = session.resource('s3')
bucket = s3.Bucket('bucketone')
for obj in bucket.objects.all():
   print(obj.key)

There is some problem with your code as well. You are creating an s3 client. S3 client does not have a Bucket method or property. To do the same thing with the s3 client you would do:

s3client = session.client('s3')
response = s3client.get_object(Bucket='bucketone', key='your key')

You can then iterate through the response that is returned to see the list of objects in the bucket.

That should take care of your error.

Answer 2

Boto3 users BEWARE

TL;DR

If you are using temporary credentials to connect to AWS services through Boto3, you MUST include a current aws_session_token as a parameter to your boto3.session.Session instance.

import os

import boto3

session = boto3.Session(
    aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    aws_session_token=os.environ["AWS_SESSION_TOKEN"],
)


# Test it on a service (yours may be different)
s3 = session.resource('s3')

# Print out bucket names
for bucket in s3.buckets.all():
    print(bucket.name)

Explanation

This is a crucial piece of information when you are testing credentials in Boto3: The error you receive may say this,

ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

but may mean you are missing an aws_session_token if you are using temporary credentials (in my case, role-based credentials).

According to AWS documentation, these are the parameters available to a boto3.session.Session object, however, there is no indication or clarification when it comes to this behavior in Boto3:

Parameters
aws_access_key_id (string) -- AWS access key ID
aws_secret_access_key (string) -- AWS secret access key
aws_session_token (string) -- AWS temporary session token
region_name (string) -- Default region when creating new connections
botocore_session (botocore.session.Session) -- Use this Botocore session instead of creating a new default one.
profile_name (string) -- The name of a profile to use. If not given, then the default profile is used.

Regarding the aws_session_token

Specifies an AWS session token used as part of the credentials to authenticate the user. A session token is required only if you manually specify temporary security credentials.

Resources

• aws_session_token
• Common scenarios for roles: Users, applications, and services
• Boto3 Credentials
• Session Reference

Answer 3

Initial answer: If you're using EC2/Lambda/etc. you'll get temporary security credentials which means you'll need to use AWS_SESSION_TOKEN environment variable and pass it to boto3.

Omitting it will throw InvalidAccessKeyId error.

Update: I can't remember exactly, but I think thr cause of my problem back then was not due to AWS_SESSION_TOKEN at all but because I bundled my own boto3 lib to Lambda app (using Zappa). After I "removed" boto3 (hence using Lambda's built-in boto3) the problem was gone.

Answer 4

You need to set up the AWS Command-line interface and configure its settings. 1. Download and install AWS CLI. 2. Run aws configure` command and provide access key and secret key. If you still get an error then check whether the IAM user's programmatic access is active or not. Delete the existing access key and create a new one for use.

Answer 5

• I faced the similar issue, Here is solution that solved my problem:

If you have saved the AWS credentials in filesystem: /.aws/credentials, You can use code like this:

s3_client = boto3.resource("s3")

• It will automatically find credentials from the file.
• Resource provide a higher-level abstraction than the raw, low-level calls made by service clients.

But, if we are using credentials in function (environment variables/program constant) use function like:

session = boto3.session.Session(
  aws_access_key_id=ACCESS_KEY_ID,
  aws_secret_access_key=SECRET_ACCESS_KEY
)
s3_client = session.client('s3')

Or

client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
)

Also, to use aws credetials in Jenkins, we can use environment variables or credentials in filesystem (file location: /var/lib/jenkins/.aws). Boto3 Documentation

Answer 6

Don't your credentials in a EC2 instance, this is not a security good practice according to AWS, instead of using aws credentials (access_key_id and secret_access_key), create an IAM role.

IAM -> Create Role -> AWS Service ( Click EC2) -> Next Permissions-> Select "AmazonS3FullAccess" or "AmazonS3ReadOnlyAccess", or whatever you prefer. After you have created, go back to your instance and attach the role name you have created to your Instance.

This will works fine.

Answer 7

Yes i know it's an old thread, but want to explain my solution. I had the same problem today with boto3.

After some researches i found that my OS time in Windows was out of sync, and because of that aws refuse or didn't recognize the key.

After synchronizing everything is ok again.

So, to be sure i made OS time out of sync, this time with purpose. And guess what; the same error.

Maybe this will be funny for someone but it's true. If you can't find the solution check your system time.