aws cli can list but boto3 cannot

Question

I can successfully perform this command so I'm pretty sure that s3:ListBucket role is provided:

aws s3 ls s3://bucket_name/folder1/folder2/
2021-09-01 21:09:36      51432 filename_2021-09-01.csv.gz
2021-09-02 21:09:32     684445 filename_2021-09-02.csv.gz
2021-09-03 21:09:34     654864 filename_2021-09-03.csv.gz
2021-09-04 21:09:31     875684 filename_2021-09-04.csv.gz

This also works fine:

aws s3 cp s3://bucket_name/folder1/folder2/filename_2021-09-01.csv.gz

I'm sure the creds in boto3 is correct because this works fine:

s3  = boto3.client('s3', region_name='us-east-1', aws_access_key_id=KEY_ID, aws_secret_access_key=ACCESS_KEY)
s3.download_file(Filename=filename,Bucket=bucket_name,Key=bucket_dir + filename)

But I cant get the list to work in boto3 without an access issue:

def keys(bucket_name, prefix='/', delimiter='/'):
    prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
    bucket = boto3.resource('s3', region_name='us-east-1', aws_access_key_id=KEY_ID, aws_secret_access_key=ACCESS_KEY).Bucket(bucket_name)
    return (_.key for _ in bucket.objects.filter(Prefix=prefix))

print(list(keys(bucket_name)))

I get the error below:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

I'm guessing my code is in correct but I've tried a few ways still Access Denied

This looks like an access issue. You should be seeing the same error if you tried an aws s3 cp for the same set of files. Looks like you only have ListBucket access granted. — vivekveeramani, Sep 10 '21 at 23:23
`aws s3 cp s3://bucket_name/folder1/folder2/filename_2021-09-01.csv.gz` works though — chowpay, Sep 11 '21 at 00:34
Are you sure both users for the `cli` and the code are the same? if you have a named profile, try explicitly specifying that in boto. — sagar1025, Sep 11 '21 at 00:54
For the record, you're using two different operations with the cli and boto3. The CLI is using `ListObjectsV2`, and as called here, boto3 is using `ListObjects`, so it is possible the permissions don't allow `ListObjects`. — Anon Coward, Sep 11 '21 at 00:58
@sagar1025 I only have 1 access/secret key to use and I used the same for both cli and boto3 — chowpay, Sep 11 '21 at 00:59
@AnonCoward you mean IAM has to have ListObjectsV2 enabled? in that case how do I get boto3 to use listobjectsV2 to match the cli? — chowpay, Sep 11 '21 at 01:01
Either test with something like `aws s3api list-objects --bucket bucket-name` vs `aws s3api list-objects-v2 --bucket bucket-name` to verify this is an issue, or call [list_objects_v2](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects_v2) on a `boto3.client('s3')` object. — Anon Coward, Sep 11 '21 at 01:06
@AnonCoward. I get Access Denied for both methods the only one that works is the one you answered in the last post `aws s3 ls s3://bucket-name-format/folder1/folder2/` aws --version shows: `aws-cli/1.20.40 Python/3.8.5 Linux/5.4.0-1032-gcp botocore/1.21.40` pip show boto3 = `Version: 1.18.40`. Also Im pretty sure I dont/wont have access to the root bucket just /folder/folder2/ that one I can list if that helps — chowpay, Sep 11 '21 at 01:37
Do you have permissions to list objects at the base bucket level? `aws s3 ls s3://bucket-name-format` — Coin Graham, Sep 12 '21 at 17:08

score 1 · Answer 1 · answered Sep 13 '21 at 09:30

From the AWS premium support webpage,

Verify that you have the permission for s3:ListBucket on the Amazon S3 buckets that you're copying objects to or from. You must have this permission to perform ListObjectsV2 actions.

Note: s3:ListBucket is the name of the permission that allows a user to list the objects in a bucket. ListObjectsV2 is the name of the API call that lists the objects in a bucket.

If your IAM user or role belong to another AWS account, then check whether your IAM and bucket policies permit the s3:ListBucket action. You must have permission to s3:ListBucket on both your IAM policy and bucket policy.

If your user or role belongs to the bucket owner's account, then you don't need both the IAM and bucket policies to allow s3:ListBucket. You need only one of them to allow the action.

Important: If either the IAM policy or the bucket policy already allow the s3:ListBucket action, then check the other policy for any statements that explicitly deny the action. An explicit deny statement overrides an allow statement.

The following is an example IAM policy that grants access to s3:ListBucket:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "Stmt1546506260896",
    "Action": "s3:ListBucket",
    "Effect": "Allow",
    "Resource": "arn:aws:s3:::AWSDOC-EXAMPLE-BUCKET"
  }]
}

client said that listing is allowed , I cant actually see their statements, to prove this they showed a `aws ls` command working as in the example above. Are you saying if aws ls works, it could still mean boto3 list bucket doesnt work? — chowpay, Sep 13 '21 at 17:33
From the article, the `s3:ListBucket` is the permission you need to use the `ListObjectsV2` which is the name of the API call that lists the objects in a bucket. — Shaqil Ismail, Sep 13 '21 at 17:43

score 0 · Accepted Answer · answered Sep 15 '21 at 17:14

Turns out I couldnt just use client. I had to make a unique session

session = boto3.Session(region_name = 'us-region', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
s3 = session.client('s3',aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)

prefix = 'folder1/fold3er2/' # cannot have a leading /
s3_result =  s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")

Print out the objects:

file_list = []
for key in s3_result['Contents']:
    file_list.append(key['Key'])
print(file_list)

aws cli can list but boto3 cannot

2 Answers2

Linked