(using aws-sdk) How to find total size of a folder stored in amazon s3

Question

I wanted to know the total size of a folder stored in S3 using AWS-SDK.

Note:-

I don't want to use any command or AWS console to find the size of my folder I wanted to do this by aws-sdk and I mentioned it above so please don't mark this as duplicate.

so far what I found on the internet is to list down all the objects of folder and iterate throw it and i do this and it's working fine. here is my code :-

import AWS from 'aws-sdk';

AWS.config.region = "BUCKET_REGION";
AWS.config.credentials = new AWS.CognitoIdentityCredentials({
   IdentityPoolId: "COGNITO_ID",
});
let bucketName = "BUCKET_NAME"
let bucket = new AWS.S3({
                params: {
                    Bucket: bucketName
                }
             });

 bucket.listObjects({Prefix:"FOLDER_NAME",Bucket:"BUCKET_NAME"}, function (err, data) {
        if (err) {
           console.log(err)                        
        } else {
           console.log(data)
           //data returns the array throw which I iterate and find the total size of the object
        }
  });

but what is the problem is that there is a point of time when my folder contains so many objects that it makes it hard to iterate each one of the elements in the list. it takes to much time to just calculate the size of the folder.

so I need a better way to calculate the size of folder and all I found is this command

aws s3 ls s3://myBucket/level1/level2/ --recursive --summarize | awk 'BEGIN{ FS= " "} /Total Size/ {print $3}'

is there any way I can do the above process throw my aws-sdk.

any kind of help is appreciated. thanks in advance

In case python is an option: https://stackoverflow.com/a/58220730/9931092 — Amit Baranes, Mar 04 '20 at 12:55
this is actually the same as what I am doing its iterate throw all the objects of the folder which I don't want to do. — Ali Asgher Badshah, Mar 04 '20 at 12:57
ok no problem let me know if there is some another way to do so — Ali Asgher Badshah, Mar 04 '20 at 12:59
Does this answer your question? [How do I find the total size of my AWS S3 storage bucket or folder?](https://stackoverflow.com/questions/32192391/how-do-i-find-the-total-size-of-my-aws-s3-storage-bucket-or-folder) — Vorsprung, Mar 04 '20 at 13:43
@Vorsprung I don't want to use AWS console or any command I wanted to do this by aws-sdk — Ali Asgher Badshah, Mar 04 '20 at 13:49
I think the third answer to that question has details on how to use the python sdk, check it out — Vorsprung, Mar 04 '20 at 13:50
thanks man for your advice but its use command-cli which is not possible for me — Ali Asgher Badshah, Mar 04 '20 at 13:52
Does this answer your question? [how to get s3 bucket size via aws-sdk (javascript or node)](https://stackoverflow.com/questions/40042150/how-to-get-s3-bucket-size-via-aws-sdk-javascript-or-node). Essentially you can't without either iterating or using cloudwatch to retrieve a value that is updated each day — Steve Mapes, Mar 04 '20 at 14:08

score 4 · Accepted Answer · answered Mar 05 '20 at 03:45

It appears that your situation is:

You want to know the size of an Amazon S3 bucket on a regular basis
The bucket contains a large number of objects, which takes too much time

Rather than listing objects and calculating sizes, I would recommend two alternatives:

Amazon S3 Inventory

Amazon S3 Inventory can provide a daily CSV file with details of all objects in a bucket. You could then take this data and calculate the total.

Amazon CloudWatch bucket metrics

Amazon CloudWatch has several metrics related to Amazon S3 buckets:

BucketSizeBytes
NumberOfObjects

I'm not sure how often those metrics are updated (they are not instant), but BucketSizeBytes seems like it would be ideal for you.

If all else fails...

If the above two options do not meet your needs (eg you need to know the metrics "right now"), the remaining option would be to maintain your own database of objects. The database would need to be updated whenever an object is added or removed from the bucket (which can be done by using Amazon S3 Events to trigger an AWS Lambda function). You could then consult your own database to have the information available rather quickly.

thanks, man I think lambda function is the most efficient way to do this. can you suggest me what is the best way although — Ali Asgher Badshah, Mar 05 '20 at 05:47
You'll have to code it yourself. Use Amazon S3 Events to trigger a Lambda function. That Lambda function would need to update a database of your choice with the objects being added/removed. The decision on an appropriate database depends on how you are going to _use_ the data. For example, it depends on whether you'll be retrieving data by user or by path, doing searches, etc. It won't be trivial, so make sure you can't just use one of the simpler options instead. — John Rotenstein, Mar 05 '20 at 05:58

Mike Bendorf · Answer 2 · 2020-12-14T18:42:13.047

This lambda method is pretty fast and it can work well for buckets with up to 100,000 objects if you are not concerned about a couple seconds delay. The AWS CLI has around the same performance because it seems to be using the same API, and S3 Metrics or Cloudwatch Stats might be more complicated to configure especially if you want to look only at specific folders.

Storing this in info in a database and triggering the method within intervals using flags is the way to go for small size buckets or folders.

const AWS = require('aws-sdk'), s3 = new AWS.S3()

exports.handler = async function (event) {
    var totalSize = 0, ContinuationToken
  do {
    var resp = await s3.listObjectsV2({
      Bucket: bucketName,
      Prefix: `folder/subfolder/`,
      ContinuationToken
    }).promise().catch(e=>console.log(e))
    resp.Contents.forEach(o=>totalSize+=o.Size)
    ContinuationToken = resp.NextContinuationToken
  } while (ContinuationToken)

  console.log(totalSize) //your answer
}

(using aws-sdk) How to find total size of a folder stored in amazon s3

2 Answers2