boto3.exceptions.S3UploadFailedError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Question

I'm running an Amazon EC2 (Ubuntu) instance which outputs a JSON file daily. I am now trying to copy this JSON to Amazon S3 so that I can eventually download it to my local machine. Following the instructions here (reading in a file from ubuntu (AWS EC2) on local machine?), I'm using boto3 to copy the JSON from Ubuntu to S3:

import boto3
print("This script uploads the SF events JSON to s3")

ACCESS_ID = 'xxxxxxxx'
ACCESS_KEY = 'xxxxxxx'
s3 = boto3.resource('s3',
         aws_access_key_id=ACCESS_ID,
         aws_secret_access_key= ACCESS_KEY)

def upload_file_to_s3(s3_path, local_path):
    bucket = s3_path.split('/')[2]
    print(bucket)
    file_path = '/'.join(s3_path.split('/')[3:])
    print(file_path)
    response = s3.Object(bucket, file_path).upload_file(local_path)
    print(response)

s3_path = "s3://mybucket/sf_events.json"
local_path = "/home/ubuntu/bandsintown/sf_events.json"
upload_file_to_s3(s3_path, local_path)

The credentials I'm using here are from creating a new user in Amazon Identity and Access Management (IAM): screenshot attached.

However, when I run this script, I get the following error:

boto3.exceptions.S3UploadFailedError: Failed to upload /home/ubuntu/bandsintown/sf_events.json to mybucket/sf_events.json: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

I've also tried attaching an IAM role to the EC2 instance and given that role full s3 permissions - but still no luck (see image below).

It appears to be a permissions issues - how might I begin to solve this? Do I need Amazon CLI? I'm also reading in boto3 documentation that I may need an aws_session_token parameter in my script.

What permissions does the IAM role have? Does it have S3:putObject permission? — Najiba Halim, Dec 08 '19 at 21:22

Arun Kamalanathan · Accepted Answer · 2019-12-08T21:42:15.157

6

Since it's an ec2, you can assign an IAM role to the instance and assign permissions to the role. Also you don't need to hardcode the credentials in your code.

https://aws.amazon.com/premiumsupport/knowledge-center/assign-iam-role-ec2-instance/

You can use this policy for S3 upload

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "statement1",
            "Effect": "Allow",
            "Action":   ["s3:PutObject","s3:PutObjectAcl"],
            "Resource": "arn:aws:s3:::examplebucket/*"
        }
    ]
}

here is how you attach policies to IAM role: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html

and change your code to:

s3 = boto3.resource('s3')

edited Dec 08 '19 at 21:42

answered Dec 08 '19 at 21:26

Arun Kamalanathan

8,107
4
23
39

Arun- see my updated question - i followed your suggestion and added a iam role for the ec2 instance but still no luck...? – DiamondJoe12 Dec 08 '19 at 21:41
did you assign policy to the IAM role – Arun Kamalanathan Dec 08 '19 at 21:42
because if you have the role and then use access keys in the code, access keys will take precedence – Arun Kamalanathan Dec 08 '19 at 21:46
I will try that, thanks! But, why isn't the AmazonS3FullAccess Policy sufficient? Why do I need to add the additional policy? – DiamondJoe12 Dec 08 '19 at 21:48
1

`AmazonS3FullAccess ` is definitely sufficient . – Arun Kamalanathan Dec 08 '19 at 21:48
if you are giving `AmazonS3FullAccess`, you don't need to assign additional policy. Usually its better to assign the minimum permission required. – Arun Kamalanathan Dec 08 '19 at 21:50
Hmm, still no luck. Now I'm getting this error: NoCredentialsError botocore.exceptions.NoCredentialsError: Unable to locate credentials – DiamondJoe12 Dec 08 '19 at 21:50
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/203860/discussion-between-arun-k-and-diamondjoe12). – Arun Kamalanathan Dec 08 '19 at 21:50
click the chat link, i will help you – Arun Kamalanathan Dec 08 '19 at 21:52
have you assigned the IAM role to the instance, can you show me a screenshot of the instance details page – Arun Kamalanathan Dec 08 '19 at 21:53
have you got it sorted out ? – Arun Kamalanathan Dec 09 '19 at 03:26

boto3.exceptions.S3UploadFailedError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

1 Answers1