Reading a file from s3 to sagemaker on AWS gives 403 forbidden error, but other operations work on the file

Question

This command:

BUCKET_TO_READ='my-bucket'
FILE_TO_READ='myFile'
data_location = 's3://{}/{}'.format(BUCKET_TO_READ, FILE_TO_READ)
df=pd.read_csv(data_location)

is failing with a

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Error and I'm unable to figure out why. That should work according to https://stackoverflow.com/a/50244897/3763782

Here are my permissions on the bucket:

            "Action": [
                "s3:ListMultipartUploadParts",
                "s3:ListBucket",
                "s3:GetObjectVersionTorrent",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectTorrent",
                "s3:GetObjectTagging",
                "s3:GetObjectAcl",
                "s3:GetObject"

And these commands work as expected:

role = get_execution_role()
region = boto3.Session().region_name
print(role)
print(region)

s3 = boto3.resource('s3')
bucket = s3.Bucket(BUCKET_TO_READ)
print(bucket.creation_date)

for my_bucket_object in bucket.objects.all():
    print(my_bucket_object)
    FILE_TO_READ = my_bucket_object.key
    break

obj = s3.Object(BUCKET_TO_READ, FILE_TO_READ)
print(obj)

All of those print statements worked just fine.

I'm not sure if it matters, but each file is within a folder, so my FILE_TO_READ looks like folder/file.

This command which should download the file to sagemaker also falied with a 403:

import boto3
s3 = boto3.resource('s3')
s3.Object(BUCKET_TO_READ, FILE_TO_READ).download_file(FILE_TO_READ)

This is also happening when I open a terminal and use

aws s3 cp AWSURI local_file_name

Answer 1

The reason was that we granted permission to the bucket not the objects. That would be granting "Resource": "arn:aws:s3:::bucket-name/" but not "Resource": "arn:aws:s3:::bucket-name/*"

Answer 2

Try differentiating the object level actions and bucket level actions in the IAM policy. Something like this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",            
            "Action":   [
                "s3:GetObjectVersionTorrent",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectTorrent",
                "s3:GetObjectTagging",
                "s3:GetObjectAcl",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::bucket-name/*"
        },
        {
            "Effect": "Allow",            
            "Action":   [
                "s3:ListMultipartUploadParts",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::bucket-name"
        }
    ]
}