0

I want to get bucket size of AWS S3 by using .net.

Want to calculated allocated space, used space for the bucket.

  1. If my bucket name is XYZ.
  2. Inside my bucket there are n numbers of subfolders.
  3. I want complete size of XYZ bucket.

for this access your have to give permission to bucket for public access through S3 browser or AWS console. Will get output like below

<Key>File Name</Key>
<LastModified>Last Modified Date</LastModified>
<ETag>Some Text</ETag>
<Size>173266</Size>

Size : in Bytes

Now I have calculate one by one file size and then to do calculations.

If any one can suggest good idea to get complete bucket size in one method that would be very good for me.

Rakesh Thakre
  • 40
  • 5
  • Cloud storage has *no folders*. Whether it's AWS S3 or Azure Blob storage, a bucket is a flat list of files. Folders are *emulated* by treating a character in the name as a separator. This doesn't even have to be `/`, that's just the default – Panagiotis Kanavos Sep 21 '22 at 05:57
  • Why do you want this in the first place? It matters. Blob storage doesn't behave the same as a folder on disk. This simply doesn't scale to buckets with millions of objects. There are no folders to begin with. Calculating the total size of a bucket is too expensive, which is why it's not updated automatically. If calculating the size of a local folder is slow, imagine how much slower it is to calculate the size of a bucket with 1M objects that are actively being updated – Panagiotis Kanavos Sep 21 '22 at 06:01
  • @PanagiotisKanavos , yes I am agreed with you. If you have any solutions for same let me know. – Rakesh Thakre Sep 21 '22 at 06:04
  • To which problem? You haven't described what the actual problem is yet. Do you want to know the size at this very moment? Or periodically monitor the size of your buckets? If periodically, would it be enough to have a (paid) AWS service like Cloudwatch do it for you? Or do you want your own on-prem service to do this? The options are described in [Find out the size of your Amazon S3 buckets](https://aws.amazon.com/blogs/storage/find-out-the-size-of-your-amazon-s3-buckets/). In all cases, something has to list all objects and sum the sizes – Panagiotis Kanavos Sep 21 '22 at 06:18
  • If you want to write your own code you'll have to use [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) to retrieve the items 1000 at a time. After each page you'll have to use the page's `NextContinuationToken` in the next request to get the next page – Panagiotis Kanavos Sep 21 '22 at 06:20
  • [This sample](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ListingKeysUsingAPIs.html) in the AWS docs shows how to retrieve all object metadata recursively. Instead of printing the keys of each response you need to sum the sizes, eg using LINQ `total+=response.S3Objects.Sum(obj=>obj.Size);` – Panagiotis Kanavos Sep 21 '22 at 06:23
  • On top of that, buckets aren't disks and have no "allocated" space. Buckets are stored in block storage arrays using compression and deduplication which means that duplicate files or even duplicate blocks in different files don't take extra space. You're billed for the reported file size though, not the actual storage. – Panagiotis Kanavos Sep 21 '22 at 06:38

2 Answers2

2

There is no API call available to obtain the size of an Amazon S3 bucket.

You can either:

  • Call ListObjects() to obtain a listing of all objects and total their sizes (note: Only 1000 objects are returned per call, so it would need to loop until all objects are returned), or
  • Use Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects. You can then calculate the total bucket size from that file.
  • Use Amazon CloudWatch Metrics that list the size of every bucket. From Monitoring metrics with Amazon CloudWatch - Amazon Simple Storage Service: "These storage metrics for Amazon S3 are reported once per day and are provided to all customers at no additional cost.

Another option (I haven't tried it) might be to use Amazon S3 Storage Lens, which can track metrics in S3.

See also:

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • I already checked this links but need to do in c#.net. – Rakesh Thakre Sep 20 '22 at 12:21
  • Additionally, S3 exposes a CloudWatch metric of [BucketSizeBytes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-dimensions.html) metric that can be called from any AWS SDK. – Anon Coward Sep 20 '22 at 13:50
  • Oh yes! There's a CloudWatch metric as mentioned by @AnonCoward that pre-computes the size of the bucket. It's a little tricky to get stats out of CloudWatch (lots of parameters!) but that would be a simple solution. I suspect that it is only updated once per day, but that might be sufficient for you. – John Rotenstein Sep 20 '22 at 21:46
  • Thanks @JohnRotenstein but do you have some examples regarding same. I am trying to solve this at my end but if you provide little portion of code that would be really appreciable – Rakesh Thakre Sep 21 '22 at 03:58
  • First, you should pick an approach. Do you need the bucket size "right now", or would a once-daily figure suffice? If CloudWatch is acceptable, then see: [Quickest way to get amazon s3 bucket size](https://stackoverflow.com/a/32179681/174777). There are API calls available from .Net to match what it is doing with the AWS CLI. – John Rotenstein Sep 21 '22 at 04:09
  • I suspect `calculated allocated space, used space for the bucket.` is significant, and the OP wants to calculate how much storage is actually billed. There's no `allocated` space as far as clients are concerned though. Whether AWS uses compression or deduplication, only the used size is reported and billed – Panagiotis Kanavos Sep 21 '22 at 06:40
0

Cloud storage has no folders. Whether it's AWS S3 or Azure Blob storage, a bucket is a flat list of files. Folders are emulated by treating a character in the name as a separator. A bucket isn't a disk either, so it has no allocated vs used space statistics. Objects are stored in blob storage arrays that use compression and deduplication.

If you want to find out what storage you're actually billed for, it's the total object size, not the actual physical storage. If you upload the same 1MB file 100 times you'll be billed for 100MB even if just 1MB are actually stored.

Calculating the total size of a bucket is too expensive, which is why it's not updated automatically. If calculating the size of a local folder is slow, imagine how much slower it is to calculate the size of a bucket with 1M objects that are actively being updated.

The only way to get the used space is to list the objects and sum their sizes. This operation is billed, no matter what tool is used to perform it.

The article Find out the size of your Amazon S3 buckets shows which tools and services can be used to monitor the actual size.

  • Amazon S3 console
  • S3 Storage Lens
  • Amazon CloudWatch
  • Amazon S3 Inventory

These services calculate and cache the size periodically, to avoid delays and extra billing.

The other option is to list and sum all objects either through the AWS CLI or code. You can use the code in Listing object keys programmatically to list all objects in a bucket one page at a time and sum the sizes:

public static async Task<long> GetBucketSizeAsync(IAmazonS3 client, string bucketName)
{
    long total=0;
    var request = new ListObjectsV2Request
    {
        BucketName = bucketName,
    };

    var response = new ListObjectsV2Response();

    do
    {
        response = await client.ListObjectsV2Async(request);

        total += response.S3Objects.Sum(obj =>obj.Size);

        request.ContinuationToken = response.NextContinuationToken;
    }
    while (response.IsTruncated);

    return total;
}

Remember, ListObjectsV2Async is billed. You should cache the results instead of calling this all the time.

Panagiotis Kanavos
  • 120,703
  • 13
  • 188
  • 236
  • Panagiotis Kanavos. Thanks a lot for help but I tried this code before its return from the below line without showing any output response = await client.ListObjectsV2Async(request); – Rakesh Thakre Sep 21 '22 at 07:14
  • The method always returns a response or an exception. If you have a problem you'll have to post the *actual* code you used and explain what `without showing any output` means. If the *method* was buggy every .NET developer using AWS would notice. Have you tried debugging the code? Is `S3Objects` empty? Do you have adequate permissions to this bucket? – Panagiotis Kanavos Sep 21 '22 at 07:24
  • Panagiotis Kanavos already assigned all permission to bucket. Assign public access to bucket. No Response or No Exception only when come to line response = await client.ListObjectsV2Async(request). , its directly show mail form by skipping all the things. – Rakesh Thakre Sep 21 '22 at 07:33
  • 'public static async Task Main() { IAmazonS3 s3Client = new AmazonS3Client("API_KEY", "API_SECKRET_KEY",Amazon.RegionEndpoint.USWest1); Console.WriteLine($"Listing the objects contained in {BucketName}:\n"); await GetBucketSizeAsync(s3Client, BucketName); }' – Rakesh Thakre Sep 21 '22 at 07:36
  • 'The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.' Now this Exceptions throwing by code. – Rakesh Thakre Sep 21 '22 at 08:48
  • This means the code used the wrong bucket or region name. There are several SO questions about this error message [like this one](https://stackoverflow.com/questions/20137806/amazon-s3-net-sdk-the-bucket-you-are-trying-to-access-must-be-addressed-using) or [this one](https://stackoverflow.com/questions/25027462/aws-s3-the-bucket-you-are-attempting-to-access-must-be-addressed-using-the-spec) – Panagiotis Kanavos Sep 21 '22 at 08:50
  • My assumption was as per your view but I checked with all region [AWS Region https://docs.aws.amazon.com/general/latest/gr/rande.html] – Rakesh Thakre Sep 21 '22 at 09:02
  • That's meaningless. You need to use the *correct* region. All others will fail. Doesn't the exception specify the correct endpoint? – Panagiotis Kanavos Sep 21 '22 at 09:05
  • In S3 Bucket browser showing region : **Location: US East (N. Virginia) (us-east-1)** Still not showing expected results. – Rakesh Thakre Sep 21 '22 at 09:07
  • Working with it only. – Rakesh Thakre Sep 21 '22 at 09:08