How to determine s3 folder size?

Question

I have a bunch of s3 folders for different projects/clients and I would like to estimate total size (so I can for instance consider reducing sizes/cost). What is a good way to determine this?

Similar to: [How to find size of a folder inside an S3 bucket?](https://stackoverflow.com/q/49759940/174777) — John Rotenstein, Mar 30 '20 at 03:33

score 1 · Answer 1 · answered Mar 29 '20 at 09:25

1

I can do this with a combination of Python and the AWS client:

import os

bucket_rows = os.popen('aws s3 ls').split(chr(10))
sizes = dict()

for bucket in bucket_rows:
    buck = bucket.split(' ')[-1]    # the full row contains additional information
    cmd = f"aws s3 ls --summarize --human-readable --recursive s3://{buck}/ | grep 'Total'"
    sizes[buck] = os.popen(cmd).read()

answered Mar 29 '20 at 09:25

Sergio Lucero

862
1
12
21

`aws s3 ls s3://bucket/path/to/data/ --summarize --human-readable --recursive | grep 'Total'` is an excellent way to quickly get subtotal size. Thanks! – Wassadamo Nov 30 '21 at 07:56

score 1 · Answer 2 · answered Mar 29 '20 at 12:54

1

As stated here AWS CLI natively supports --query parameter with can determine the size of every object in S3 bucket.

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

I hope it helps.

answered Mar 29 '20 at 12:54

Vaibhav Jain

2,155
5
27
41

score 1 · Answer 3 · answered Feb 02 '23 at 22:05

1

This would do the magic

for bucket_name in `aws s3 ls | awk '{print $3}'`; do
  echo "$bucket_name"
  aws s3 ls s3://$bucket_name --recursive --summarize | tail -n2
done

answered Feb 02 '23 at 22:05

Jorge Tovar

1,374
12
17

score 0 · Answer 4 · answered Mar 29 '20 at 19:26

If you want to check via console.

If you mention a folder by folder and not bucket, then just select that object go to "action" drop down and select "Get total size"
If you mean bucket by folder , then go to management tab and in that go to metrics it will show entire bucket size

How to determine s3 folder size?

4 Answers4