AccessDenied for ListObjects for S3 bucket when permissions are s3:*

Question

I am getting:

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

When I try to get folder from my S3 bucket.

Using this command:

aws s3 cp s3://bucket-name/data/all-data/ . --recursive

The IAM permissions for the bucket look like this:

{
"Version": "version_id",
"Statement": [
    {
        "Sid": "some_id",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::bucketname/*"
        ]
    }
] }

What do I need to change to be able to copy and ls successfully?

In my case, I had configured `aws` for one user and used it inside a cronjob-called bash script from another user, meaning the access key and access token were wrong/unset. My solution was to directly put the credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`) into my bash script file [as described here](https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-environment.html). — Uwe Keim, Apr 25 '18 at 05:49

score 360 · Accepted Answer · edited May 19 '20 at 19:10

360

You have given permission to perform commands on objects inside the S3 bucket, but you have not given permission to perform any actions on the bucket itself.

Slightly modifying your policy would look like this:

{
  "Version": "version_id",
  "Statement": [
    {
        "Sid": "some_id",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::bucketname",
            "arn:aws:s3:::bucketname/*"
        ]
    }
  ] 
}

However, that probably gives more permission than is needed. Following the AWS IAM best practice of Granting Least Privilege would look something like this:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::bucketname"
          ]
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject"
          ],
          "Resource": [
              "arn:aws:s3:::bucketname/*"
          ]
      }
  ]
}

edited May 19 '20 at 19:10

Robin Métral

3,099
3
17
32

answered Aug 04 '16 at 19:04

Mark B

183,023
24
297
295

48

This is too wide an actions list. For list and copy rights, what would you suggest are the actions required. – dom farr Mar 13 '17 at 14:23
9

@domfarr I was just doing the minimal possible to adjust the OPs policy to make it work for him. If you have a specific question you should post it as a separate question on this site instead of tagging onto this one with comments. Although it should be really easy to look at the list of S3 actions and build the policy you want. – Mark B Mar 13 '17 at 14:34
6

I wasn't tagging. OP did request what was needed for copy and ls. The lack of bucket level arn was included, but you didn't adjust the Actions...hence my comment. – dom farr Mar 19 '17 at 19:26
1

Maybe directly assigning an appropriate role to EC3 should work. :) – ChikuMiku May 27 '18 at 23:29
14

Also be sure not to write `"Resource": [ "arn:aws:s3:::bucketname/", "arn:aws:s3:::bucketname/*" ]` like I did (i.e. you only want a slash before the `*`) – took me 4 long hours to realize this mistake caused all my `listObjects` calls to fail... – Dániel Kis-Nagy Sep 12 '18 at 14:18
Doing that you give access to make wathever. It isn't safe. – Carlos Andres Feb 25 '19 at 18:14
1

@CarlosAndres how "safe" it is depends entirely on what this IAM policy is being assigned to, and if the bucket is open to the public or not. Please feel free to provide your own answer with the permissions you think are more appropriate. – Mark B Feb 25 '19 at 18:23
It really should not be suggested to add Action: s3:* to resolve this problem. The permissions will be way too wide and might cause security issues. – Tomas Vinter Oct 02 '19 at 13:36
@TomasVinter you are repeating other comments here. Please see the discussion already in the comments. – Mark B Oct 02 '19 at 14:19
I get the impression that if you do `aws s3 ls` without noting a particular bucket, you may struggle for a while. – mlissner Oct 23 '19 at 20:17
It took me some time to figure this out that there has to be separate statement, if you combine both, it will throw same access denied error – Khuram Niaz Oct 16 '22 at 22:43

Robert Smith · Answer 2 · 2019-10-03T16:21:16.023

If you wanted to copy all s3 bucket objects using the command "aws s3 cp s3://bucket-name/data/all-data/ . --recursive" as you mentioned, here is a safe and minimal policy to do that:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "s3:ListBucket"
          ],
          "Resource": [
              "arn:aws:s3:::bucket-name"
          ],
          "Condition": {
              "StringLike": {
                  "s3:prefix": "data/all-data/*"
              }
          }
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject"
          ],
          "Resource": [
              "arn:aws:s3:::bucket-name/data/all-data/*"
          ]
      }
  ]
}

The first statement in this policy allows for listing objects inside a specific bucket's sub directory. The resource needs to be the arn of the S3 bucket, and to limit listing to only a sub-directory in that bucket you can edit the "s3:prefix" value.

The second statement in this policy allows for getting objects inside of the bucket at a specific sub-directory. This means that anything inside the "s3://bucket-name/data/all-data/" path you will be able to copy. Be aware that this doesn't allow you to copy from parent paths such as "s3://bucket-name/data/".

This solution is specific to limiting use for AWS CLI commands; if you need to limit S3 access through the AWS console or API, then more policies will be needed. I suggest taking a look here: https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/.

A similar issue to this can be found here which led me to the solution I am giving. https://github.com/aws/aws-cli/issues/2408

Hope this helps!

It is indeed possible, https://aws.amazon.com/blogs/security/writing-iam-policies-grant-access-to-user-specific-folders-in-an-amazon-s3-bucket/ — citynorman, Dec 29 '18 at 03:21
I've edited the answer to include a condition to limit listing objects which are under a specific sub-folder. Thanks for the help! — Robert Smith, Oct 03 '19 at 16:22
Do pay attention to the s3:prefix in the example. I couldn't get a similar policy working and I'd missed the '/*' off the end of the s3:prefix. Without this I could list the bucket, but s3 sync and s3 cp didn't work. — biddster, Nov 13 '19 at 11:15
@biddster That's correct, the '/*' acts a wildcard to match any objects in the specified s3 subdirectory "/data/all-data". Thanks for pointing that out. :) — Robert Smith, Nov 13 '19 at 18:06

score 10 · Answer 3 · answered Jul 03 '17 at 04:33

I got the same error when using policy as below, although i have "s3:ListBucket" for s3:ListObjects operation.

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl"
        ],
        "Resource": [
            "arn:aws:s3:::<bucketname>/*",
            "arn:aws:s3:::*-bucket/*"
        ],
        "Effect": "Allow"
    }
  ]
 }

Then i fixed it by adding one line "arn:aws:s3:::bucketname"

{
"Version": "2012-10-17",
"Statement": [
    {
        "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl"
        ],
        "Resource": [
             "arn:aws:s3:::<bucketname>",
            "arn:aws:s3:::<bucketname>/*",
            "arn:aws:s3:::*-bucket/*"
        ],
        "Effect": "Allow"
    }
 ]
}

This was my problem Wildcard on the bucket didn't work for ListBucket. I had to specify the exact bucket name. — Doug Lampe, Aug 30 '20 at 17:56

score 9 · Answer 4 · answered Aug 01 '19 at 23:24

9

I tried the following:

aws s3 ls s3.console.aws.amazon.com/s3/buckets/{bucket name}

This gave me the error:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Using this form worked:

aws s3 ls {bucket name}

answered Aug 01 '19 at 23:24

Henry

7,721
2
38
38

not passing the `bucket name` is what I was missing, thank you. – darul75 Dec 19 '22 at 21:07

score 9 · Answer 5 · answered Mar 03 '22 at 07:57

9

For Amazon users who have enabled MFA, please use this: aws s3 ls s3://bucket-name --profile mfa.

And prepare the profile mfa first by running aws sts get-session-token --serial-number arn:aws:iam::123456789012:mfa/user-name --token-code 928371 --duration 129600. (replace 123456789012, user-name and 928371).

answered Mar 03 '22 at 07:57

Lane

4,682
1
36
20

THIS is correct answer, already spent 2 days man, thank you very much! – kokemomuke Mar 22 '22 at 13:09

score 8 · Answer 6 · answered Mar 30 '17 at 20:05

8

I was unable to access to S3 because

first I configured key access on the instance (it was impossible to attach role after the launch then)
forgot about it for a few months
attached role to instance
tried to access. The configured key had higher priority than role, and access was denied because the user wasn't granted with necessary S3 permissions.

Solution: rm -rf .aws/credentials, then aws uses role.

answered Mar 30 '17 at 20:05

Putnik

5,925
7
38
58

1

Exact same problem. A symptom of this will be one user can access things that another user cannot on the same EC2 node. – Doc Kaos Sep 05 '17 at 19:04

Ihor Pavlyk · Answer 7 · 2019-07-30T16:03:43.710

I faced with the same issue. I just added credentials config:

aws_access_key_id = your_aws_access_key_id
aws_secret_access_key = your_aws_secret_access_key

into "~/.aws/credentials" + restart terminal for default profile.

In the case of multi profiles --profile arg needs to be added:

aws s3 sync ./localDir s3://bucketName --profile=${PROFILE_NAME}

where PROFILE_NAME:

.bash_profile ( or .bashrc) -> export PROFILE_NAME="yourProfileName"

More info about how to config credentials and multi profiles can be found here

Bonny James · Answer 8 · 2021-09-24T09:44:00.367

Okay for those who have done all the above and still getting this issue, try this:

Bucket Policy should look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowBucketSync",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

Then save and ensure your Instance or Lightsail is connected to the right profile on AWS Configure.

First: try adding --recursive at the end, any luck? No okay try the one below.

Second: Okay now try this instead: --no-sign-request

so it should look like this:

sudo aws s3 sync s3://BUCKET_NAME /yourpath/path/folder --no-sign-request

You're welcome

score 5 · Answer 9 · edited Nov 03 '20 at 18:19

5

You have to specify Resource for the bucket via "arn:aws:s3:::bucketname" or "arn:aws:3:::bucketname*". The latter is preferred since it allows manipulations on the bucket's objects too. Notice there is no slash!

Listing objects is an operation on Bucket. Therefore, action "s3:ListBucket" is required. Adding an object to the Bucket is an operation on Object. Therefore, action "s3:PutObject" is needed. Certainly, you may want to add other actions as you require.

{
"Version": "version_id",
"Statement": [
    {
        "Sid": "some_id",
        "Effect": "Allow",
        "Action": [
            "s3:ListBucket",
            "s3:PutObject"
        ],
        "Resource": [
            "arn:aws:s3:::bucketname*"
        ]
    }
] 
}

edited Nov 03 '20 at 18:19

Robert Johnstone

5,431
12
58
88

answered Apr 05 '17 at 12:42

marzhaev

479
4
4

30

the problem with `arn:aws:s3:::bucketname*` is that it also allows access to `arn:aws:s3:::bucketname-with-suffix` – Guss Sep 12 '17 at 16:23
3

As written, this is a bad policy; don't use it. It should allow s3:ListBucket against arn:aws:s3:::bucketname and s3:PutObject against arn:aws:s3:::bucketname/* – jarmod Oct 03 '19 at 13:22

score 4 · Answer 10 · answered Dec 14 '18 at 01:44

4

I was thinking the error is due to "s3:ListObjects" action but I had to add the action "s3:ListBucket" to solve the issue "AccessDenied for ListObjects for S3 bucket"

answered Dec 14 '18 at 01:44

Sudhakar Naidu

244
2
7

score 3 · Answer 11 · answered May 28 '20 at 00:42

I'm adding an answer with the same direction as the accepted answer but with small (important) differences and adding more details.

Consider the configuration below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::<Bucket-Name>"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::<Bucket-Name>/*"]
    }
  ]
}

The policy grants programmatic write-delete access and is separated into two parts:
The ListBucket action provides permissions on the bucket level and the other PutObject/DeleteObject actions require permissions on the objects inside the bucket.

The first Resource element specifies arn:aws:s3:::<Bucket-Name> for the ListBucket action so that applications can list all objects in the bucket.

The second Resource element specifies arn:aws:s3:::<Bucket-Name>/* for the PutObject, and DeletObject actions so that applications can write or delete any objects in the bucket.

The separation into two different 'arns' is important from security reasons in order to specify bucket-level and object-level fine grained permissions.

Notice that if I would have specified just GetObject in the 2nd block what would happen is that in cases of programmatic access I would receive an error like:

Upload failed: <file-name> to <bucket-name>:<path-in-bucket> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.

score 2 · Answer 12 · answered Oct 04 '21 at 12:45

I had a similar problem while trying to sync an entire s3 bucket locally. For me MFA (Multi-factor authentication) was enforced on my account, which is required while making commands via AWS CLI.

So the solution for me was - provide mfa credentials using a profile (mfa documentation) while using any AWS CLI commands.

score 1 · Answer 13 · answered Dec 01 '20 at 18:05

1

To allow permissions in s3 bucket go to the permissions tab in s3 bucket and in bucket policy change the action to this which will allow all actions to be performed:

"Action":"*"

answered Dec 01 '20 at 18:05

SYED FAISAL

495
5
8

score 1 · Answer 14 · answered Jul 13 '21 at 00:35

Here's the policy that worked for me.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::bucket-name/*"
      ]
    }
  ]
}

score 1 · Answer 15 · answered Feb 12 '23 at 06:04

A little late but might be helpful for someone. First thing first I am managing all access to s3 buckets using bucket policy.

My bucket policy to allow access to folder1 to IAM user: user1

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789012:user/user1"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::s3-bucket/folder1",
                "arn:aws:s3:::s3-bucket/folder1/*"
            ]
        }
    ]
}

Now when user1 tries to perform list operation they get an error. It may look weird as the user has s3 full access from the bucket policy.

aws s3 ls s3://s3-bucket/folder1
aws s3 ls s3://s3-bucket/folder1/
aws s3 ls s3://s3-bucket/folder1/*

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Now let's take a look at the AWS documentation for ListBucket

Grants permission to list some or all of the objects in an Amazon S3 bucket (up to 1000)

To test that try to create a bucket policy and only provide the ListBucket permission for folder1 like this. Observe that you will get an error.

Conclusion
The ListBucket operation is only permitted for buckets and not for prefixes and hence if we want to provide list operation then it must be at the bucket level. Of course this will allow the user to list objects inside all other folders present in the bucket.

score 0 · Answer 16 · answered Aug 01 '19 at 22:15

Ran into a similar issues, for me the problem was that I had different AWS keys set in my bash_profile.

I answered a similar question here: https://stackoverflow.com/a/57317494/11871462

If you have conflicting AWS keys in your bash_profile, AWS CLI defaults to these instead.

score 0 · Answer 17 · answered Feb 02 '20 at 20:59

I had this issue my requirement i wanted to allow user to write to specific path

{
            "Sid": "raspiiotallowspecificBucket",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>/scripts",
                "arn:aws:s3:::<bucketname>/scripts/*"
            ]
        },

and problem was solved with this change

{
            "Sid": "raspiiotallowspecificBucket",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>",
                "arn:aws:s3:::<bucketname>/*"
            ]
        },

BlackMetalOwl · Answer 18 · 2020-08-19T18:31:37.960

I like this better than any of the previous answers. It shows how to use the YAML format and lets you use a variable to specify the bucket.

    - PolicyName: "AllowIncomingBucket"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Action: "s3:*"
            Resource:
              - !Ref S3BucketArn
              - !Join ["/", [!Ref S3BucketArn, '*']]

score 0 · Answer 19 · answered Mar 13 '21 at 00:15

My issue was having set

env: 
  AWS_ACCESS_KEY_ID: {{ secrets.AWS_ACCESS_KEY_ID }} 
  AWS_SECRET_ACCESS_KEY: {{ secrets.AWS_SECRET_ACCESS_KEY }}

again, under the aws-sync GitHub Action as environment variables. They were coming from my GitHub settings. Though in my case I had assumed a role in the previous step which would set me some new keys into those same name environment variables. So i was overwriting the good assumed keys with the bad GitHub basic keys.

Please take care of this if you're assuming roles.

score 0 · Answer 20 · answered May 11 '21 at 21:03

I had the same issue. I had to provide the right resource and action, resource is your bucket's arn and action in your desired permission. Also please ensure you have your right user arn. Below is my solution.

{
    "Version": "2012-10-17",
    "Id": "Policy1546414123454",
    "Statement": [
        {
            "Sid": "Stmt1546414471931",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789101:root"
            },
            "Action": ["s3:ListBucket", "s3:ListBucketVersions"],
            "Resource": "arn:aws:s3:::bucket-name"
        }
    ]
}

score 0 · Answer 21 · answered Jun 28 '22 at 09:52

If you are suddenly getting this error on a new version of minio on buckets that used to work, the reason is that bucket access policy defaults were changed from version 2021 to 2022. Now in version 2022 by default all buckets (both newly created and existing ones) have Access Policy set to Private - it is not sufficient to provide server credentials to access them - you will still get errors such as these (here: returned to the python minio client):

S3Error: S3 operation failed; code: AccessDenied, message: Access Denied., resource: /dicts, request_id: 16FCBE6EC0E70439, host_id: 61486e5a-20be-42fc-bd5b-7f2093494367, bucket_name: dicts

To roll back to the previous security settings in version 2022, the quickest method is to change the bucket access Access Policy back to Public in the MinIO console (or via mc client).

score 0 · Answer 22 · answered Sep 24 '22 at 05:42

0

This is not the best practice but this will unblock you. Make sure for the user that's executing the command, it has the following policy attached to it under it's permission. A. PowerUserAccess B. AmazonS3FullAccess

answered Sep 24 '22 at 05:42

grepit

21,260
6
105
81

Manikandan Jayaraman · Answer 23 · 2023-01-13T09:06:01.870

I had faced same error "An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied"

Note: Bucket policy not a good solution. In IAM service create new custom policy attached with respective user would be safer.

Solved by below procedure:

IAM Service > Policies > Create Policy > select JSON >

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:ListBucketVersions"
        ],
        "Resource": [
            "arn:aws:s3:::<bucket name>"
        ]
    },
    {
        "Effect": "Allow",
        "Action": [
            "s3:ListBucket",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:ListBucketMultipartUploads",
            "s3:ListMultipartUploadParts",
            "s3:AbortMultipartUpload",
            "s3:DeleteObjectVersion",
            "s3:GetObjectVersion",
            "s3:PutObjectACL",
            "s3:ListBucketVersions"
        ],
        "Resource": [
            "arn:aws:s3:::<bucketname>/*"
        ]
    }
]

}

Select Next Tag > Review Policy enter and create policy.

Select the newly created policy Select the tab 'Policy Usage' in edit window of newly created policy window. Select "Attach" select the user from the list and Save.

Now try in console with bucket name to list the objects, without bucket name it throws same error.

$aws s3 ls

score 0 · Answer 24 · edited May 31 '23 at 12:46

0

I managed to solve it by changing the policy on the VPC endpoint.

Sample1

Sample2

edited May 31 '23 at 12:46

4b0

21,981
30
95
142

answered May 31 '23 at 12:37

Jimmy

9
1

AccessDenied for ListObjects for S3 bucket when permissions are s3:*

24 Answers24

Linked