Get Storage used by bucket in amazon s3 using C# SDK

Question

I have the task of providing an api endpoint to find out how much space a particular module is using in our Amazon s3 bucket. I'm using the C# SDK.

I have accomplished this by adapting code from the documentation here: https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingNetSDK.html

   private long GetUsedBytes(string module, string customer)
    {
        ListObjectsRequest listRequest = new ListObjectsRequest()
        {
            BucketName = BucketName,
            Prefix = module + "/" + customer
        };
        ListObjectsResponse listResponse;
        long totalSize = 0;
        do
        {
            listResponse = s3Client.ListObjects(listRequest);
            foreach (S3Object obj in listResponse.S3Objects)
            {
                totalSize += obj.Size;
            }
            listRequest.Marker = listResponse.NextMarker;
        } while (listResponse.IsTruncated);
        return totalSize;
    }

My question is: is there a way to do this with the sdk without pulling all of the actual s3objects down off the bucket? There are several good answers about doing it with the CLI:

AWS S3: how do I see how much disk space is using https://serverfault.com/questions/84815/how-can-i-get-the-size-of-an-amazon-s3-bucket

But I've yet to be able to find one using the SDK directly. Do I have to mimic the sdk somehow to accomplish this? Another method I considered is to get all the keys and query for their metadata, but the only way to get all the keys I've found is to grab all the objects as in the link above ^. If there's a way to get all the metadata for objects with a particular prefix that would be ideal.

Thanks for your time!

~Josh

Answer 1

Your code is not downloading any objects from Amazon S3. It is merely calling ListObjects() and totalling the size of each object. It will make one API call per 1000 objects.

Alternatively, you can retrieve the size of each bucket from Amazon CloudWatch.

From Monitoring Metrics with Amazon CloudWatch - Amazon S3:

Metric: BucketSizeBytes

The amount of data in bytes stored in a bucket. This value is calculated by summing the size of all objects in the bucket (both current and noncurrent objects), including the size of all parts for all incomplete multipart uploads to the bucket.

So, simply retrieve the metric from Amazon CloudWatch rather than calculating it yourself.