S3 permissions required to get bucket size?

Question

I'm using boto3 to get the size of all objects in S3 and have granted the following permissions:

s3:ListAllMyBuckets
s3:ListObjects
s3:GetObject

However boto keeps throwing this error:

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

I couldn't find any details in the docs or by looking at the source code for boto... does anyone know th e minimum permissions necessary just to get the size of all objects in an S3 bucket?

Can you share the original policy? What resource(s) was ListObjects permitted against? — jarmod, Oct 02 '19 at 22:00
Can you share a minimal example of your boto3 script? I'm intrigued and would like to understand what's really going on here. It seems odd that the error message would indicate that ListObjects was denied yet adding ListJobs permission corrected it. — jarmod, Oct 03 '19 at 12:39

Amit Baranes · Accepted Answer · 2019-10-10T10:29:19.740

I created the following lambda which prints each object size and sums up the total bucket size.

I use convert_size function from here. Credit to @James Sapam.

Code snippet :

import boto3
import math

def convert_size(size_bytes):
   if size_bytes == 0:
       return "0B"
   size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
   i = int(math.floor(math.log(size_bytes, 1024)))
   p = math.pow(1024, i)
   s = round(size_bytes / p, 2)
   return "%s %s" % (s, size_name[i])

bucket_name = 'BUCKET_NAME'
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)

def lambda_handler(event, context):
    all_objects = bucket.objects.all()
    total_size = 0
    for obj in all_objects:
        if obj.key.split('/')[-1]:
            file_name = obj.key
            file_size = convert_size(obj.size)
            total_size += obj.size
            print("File Name: %s File Size: %s" % (file_name,file_size) ) 
    print("%s bucket size : %s" % (bucket_name,convert_size(total_size)) )

Policy summary JSON :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::${BUCKET_NAME}"
        }
    ]
}

Output :

If after trying the above solution you still find issues, take a look at this thread.

S3 permissions required to get bucket size?

1 Answers1

Linked