How can I enable an ec2 instance to have private access to an S3 bucket?

Question

First of all i'm aware of these questions:

but the solutions are not working for me.

I created a role "sample_role", attached the AmazonS3FullAccess-policy to it and assigned the role to the ec2-instance.

My bucket-policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::My-Account-ID:role/sample_role"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::my_bucket/*"
        }
    ]
}

On my ec2-instance, listing my buckets works fine, both from the command line (aws s3 ls) and from python script. But when I try to upload a file test.txt to my bucket, I get AccessDenied:

import boto3

s3_client = boto3.client('s3')
s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket('my_bucket')

with open('test.txt', "rb") as f:
    s3_client.upload_fileobj(f, bucket.name, 'text.txt')

Error message:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Same happens when i just try to list the objects in my bucket. Command line aws s3api list-objects --my_bucket or python script:

import boto3

s3_resource = boto3.resource('s3')
bucket = s3_resource.Bucket('my_bucket')

for my_bucket_object in bucket.objects.all():
    print(my_bucket_object)

Error message:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

When I turn off "Block all public access" in my bucket settings and enable public access in my access control list, it obviously works. But I need to restrict access to the specified role.

What am I missing? Thanks for your help!

s3:ListBucket is a bucket-level action. It needs a resource of `arn:aws:s3:::my_bucket` (not `arn:aws:s3:::my_bucket/*` which pertains to objects). — jarmod, Dec 22 '19 at 16:56
Thanks. So i added `my_bucket/*` to my resources like: `"Resource": [ "arn:aws:s3:::my_bucket", "arn:aws:s3:::my_bucket/*" ]`, but listing the objects still fails. Also this is not explaning why the upload fails, right? — David Salb, Dec 22 '19 at 17:19
Stepping back, what are you trying to achieve with a policy on the S3 bucket? If you give an IAM role access to the bucket then you don't also need to allow it in the S3 bucket policy (typically you would have no bucket policy). Are you trying to restrict access to that bucket to a specific IAM role? — jarmod, Dec 22 '19 at 17:38
Yes i want to restrict access to the bucket to the IAM role speciefied in my bucket policy like so `"Principal": { "AWS": "arn:aws:iam::My-Account-ID:role/sample_role" }` — David Salb, Dec 22 '19 at 17:50
There's an article on this topic at https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/ — jarmod, Dec 22 '19 at 17:58
putting it simple, it would be enough to have a way to access my bucket from the ec2-instance without making it publicly available. — David Salb, Dec 22 '19 at 18:04
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/204684/discussion-between-jarmod-and-david-salb). — jarmod, Dec 22 '19 at 18:56

John Rotenstein · Answer 1 · 2019-12-23T20:32:41.887

It appears that your requirement is:

You have an Amazon S3 bucket (my_bucket)
You have an Amazon EC2 instance with an IAM Role attached
You want to allow applications running on that EC2 instance to access my_bucket
You do not want the bucket to be publicly accessible

I will also assume that you are not trying to deny other users access to the bucket if they have already been granted that access. You are purely wanting to Allow access to the EC2 instance, without needing to Deny access to other users/roles that might also have access.

You can do this by adding a policy to the IAM Role that is attached to the EC2 instance:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::my_bucket",
                "arn:aws:s3:::my_bucket/*"
            ]
        }
    ]
}

This grants ALL Amazon S3 permissions to the IAM Role for my_bucket. Note that some commands require permission on the bucket itself, while other commands require permission on the contents (/*) of the bucket.

I should also mention that granting s3:* is probably too generous, because it would allow the applications running on the instance to delete content and even delete the bucket, which is probably not what you wish to grant. If possible, limit the actions to only those that are necessary.

This gives me `Has prohibited field Principal`. However in the bucket policy `Principal` seems to be allowed. — David Salb, Dec 23 '19 at 15:05
Oops! Removed the role (not required within an IAM User/Role policy). — John Rotenstein, Dec 23 '19 at 20:33

score 1 · Answer 2 · answered Dec 23 '19 at 21:03

When I turn off "Block all public access" in my bucket settings and enable public access in my access control list, it obviously works.

Remove "enable public access" from this sentence and this will be your solution :-)

"Block all public access" blocks all public access and it doesn't matter what bucket policy you use. So uncheck this option and your bucket policy will start working as you planned.

score 1 · Accepted Answer · answered Dec 24 '19 at 15:44

So I found the problem. The credentails of my ec2 instance were configured with the access key of a dev-user account to which the role was not assigned.

I found out by running aws sts get-caller-identity which returns the identity (e.g. IAM role) actually being used.

So it seems that the assigned role can be overwritten by the user identity, which makes sense. To solve the problem, I simply undid the configuration by deleting the configuration file ~/.aws/credentials. After that the identity changed to the assigned role.

How can I enable an ec2 instance to have private access to an S3 bucket?

3 Answers3