How do I find the total size of my AWS S3 storage bucket or folder?

Question

Does Amazon provide an easy way to see how much storage my S3 bucket or folder is using? This is so I can calculate my costs, etc.

Answer 1

Two ways,

Using aws cli

aws s3 ls --summarize --human-readable --recursive s3://bucket/folder/*

If we omit / in the end, it will get all the folders starting with your folder name and give a total size of all.

aws s3 ls --summarize --human-readable --recursive s3://bucket/folder

Using boto3 api

import boto3

def get_folder_size(bucket, prefix):
    total_size = 0
    for obj in boto3.resource('s3').Bucket(bucket).objects.filter(Prefix=prefix):
        total_size += obj.size
    return total_size

Answer 2

Amazon has changed the Web interface so now you have the "Get Size" under the "More" menu.

Answer 3

Answer updated for 2021 :)

In your AWS console, under S3 buckets, find bucket, or folder inside it, and click Calculate total size.

Answer 4

As of the 28th July 2015 you can get this information via CloudWatch.

aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time 2015-07-15T10:00:00 
--end-time 2015-07-31T01:00:00 --period 86400 --statistics Average --region us-east-1 
--metric-name BucketSizeBytes --dimensions Name=BucketName,Value=myBucketNameGoesHere 
Name=StorageType,Value=StandardStorage


Important: You must specify both StorageType and BucketName in the dimensions argument otherwise you will get no results.

Answer 5

in case if someone needs the bytes precision:

aws s3 ls --summarize --recursive s3://path | tail -1 | awk '{print $3}'

Answer 6

Answer adjusted to 2020: Go into your bucket, select all folders, files and click on "Actions"->"Get Total Size"

Answer 7

I use s3cmd du s3://BUCKET/ --human-readable to view size of folders in S3. It gives quite a detailed info about the total objects in the bucket and its size in a very readable form.

Answer 8

Using the AWS Web Console and Cloudwatch:

1. Go to CloudWatch
2. Clcik Metrics from the left side of the screen
3. Click S3
4. Click Storage

5. You will see a list of all buckets. Note there are two possible points of confusion here:

   a. You will only see buckets that have at least one object in the bucket.
   b. You may not see buckets created in a different region and you might need to switch regions using the pull down at the top right to see the additional buckets

6. Search for the word "StandardStorage" in the area stating "Search for any metric, dimension or resource id"

7. Select the buckets (or all buckets with the checkbox at the left below the word "All") you would like to calculate total size for
8. Select at least 3d (3 days) or longer from the time bar towards the top right of the screen

You will now see a graph displaying the daily (or other unit) size of list of all selected buckets over the selected time period.

Answer 9

The most recent and the easiest way is to go to "Metric" tab. It provides clear understanding of the bucket size and number of objects inside it.

Answer 10

If you don't need an exact byte count or if the bucket is really large (in the TBs or millions of objects), using CloudWatch metrics is the fastest way as it doesn't require iterating through all the objects, which can take significant CPU and can end in a timeout or network error if using a CLI command.

Based on some examples from others on SO for running the aws cloudwatch get-metric-statistics command, I've wrapped it up in a useful Bash function that allows you to optionally specify a profile for the aws command:

# print S3 bucket size and count
# usage: bsize <bucket> [profile]
function bsize() (
  bucket=$1 profile=${2-default}

  if [[ -z "$bucket" ]]; then
    echo >&2 "bsize <bucket> [profile]"
    return 1
  fi

  # ensure aws/jq/numfmt are installed
  for bin in aws jq numfmt; do
    if ! hash $bin 2> /dev/null; then
      echo >&2 "Please install \"$_\" first!"
      return 1
    fi
  done

  # get bucket region
  region=$(aws --profile $profile s3api get-bucket-location --bucket $bucket 2> /dev/null | jq -r '.LocationConstraint // "us-east-1"')
  if [[ -z "$region" ]]; then
    echo >&2 "Invalid bucket/profile name!"
    return 1
  fi

  # get storage class (assumes
  # all objects in same class)
  sclass=$(aws --profile $profile s3api list-objects --bucket $bucket --max-items=1 2> /dev/null | jq -r '.Contents[].StorageClass // "STANDARD"')
  case $sclass in
    REDUCED_REDUNDANCY) sclass="ReducedRedundancyStorage" ;;
    GLACIER)            sclass="GlacierStorage" ;;
    DEEP_ARCHIVE)       sclass="DeepArchiveStorage" ;;
    *)                  sclass="StandardStorage" ;;
  esac

  # _bsize <metric> <stype>
  _bsize() {
    metric=$1 stype=$2
    utnow=$(date +%s)
    aws --profile $profile cloudwatch get-metric-statistics --namespace AWS/S3 --start-time "$(echo "$utnow - 604800" | bc)" --end-time "$utnow" --period 604800 --statistics Average --region $region --metric-name $metric --dimensions Name=BucketName,Value="$bucket" Name=StorageType,Value="$stype" 2> /dev/null | jq -r '.Datapoints[].Average'
  }

  # _print <number> <units> <format> [suffix]
  _print() {
    number=$1 units=$2 format=$3 suffix=$4
    if [[ -n "$number" ]]; then
      numfmt --to="$units" --suffix="$suffix" --format="$format" $number | sed -En 's/([^0-9]+)$/ \1/p'
    fi
  }
  _print "$(_bsize BucketSizeBytes $sclass)" iec-i "%10.2f" B
  _print "$(_bsize NumberOfObjects AllStorageTypes)" si "%8.2f"
)

A few caveats:

• For simplicity, the function assumes that all objects in the bucket are in the same storage class!
• On macOS, use gnumfmt instead of numfmt.
• If numfmt complains about invalid --format option, upgrade GNU coreutils for floating-point precision support.

Answer 11

As an alternative, you can try s3cmd, which has a du command like Unix.

Answer 12

s3cmd du --human-readable --recursive s3://Bucket_Name/

Answer 13

There are many ways to calculate the total size of folders in the bucket

Using AWS Console

S3 Buckets > #Bucket > #folder > Actions > Calculate total size

Using AWS CLI

aws s3 ls s3://YOUR_BUCKET/YOUR_FOLDER/ --recursive --human-readable --summarize

The command's output shows:

1. The date the objects were created
2. Individual file size of each object
3. The path of each object the total number of objects in the s3 bucket
4. The total size of the objects in the bucket

Using Bash script

    #!/bin/bash
    while IFS= read -r line; 
    do
    echo $line
    aws s3 ls  --summarize  --human-readable  --recursive s3://#bucket/$line --region #region | tail -n 2 | awk '{print $1 $2 $3 $4}'
    echo "----------"
    done < folder-name.txt

Sample Output:

test1/
TotalObjects:10
TotalSize:2.1KiB
----------
s3folder1/
TotalObjects:2
TotalSize:18.2KiB
----------
testfolder/
TotalObjects:1
TotalSize:112 Mib
----------

Answer 14

Found here

aws s3api list-objects --bucket cyclops-images --output json --query "[sum(Contents[].Size), length(Contents[])]" | awk 'NR!=2 {print $0;next} NR==2 {print $0/1024/1024/1024" GB"}'

Answer 15

You can visit this URL to see the size of your bucket on the "Metrics" tab in S3: https://s3.console.aws.amazon.com/s3/buckets/{YOUR_BUCKET_NAME}?region={YOUR_REGION}&tab=metrics

The data's actually in CloudWatch so you can just go straight there instead and then save the buckets you're interested in to a dashboard.

Answer 16

In NodeJs

const getAllFileList = (s3bucket, prefix = null, token = null, files = []) => {
  var opts = { Bucket: s3bucket, Prefix: prefix };
  let s3 = awshelper.getS3Instance();
  if (token) opts.ContinuationToken = token;
  return new Promise(function (resolve, reject) {
    s3.listObjectsV2(opts, async (err, data) => {
      files = files.concat(data.Contents);
      if (data.IsTruncated) {
        resolve(
          await getAllFileList(
            s3bucket,
            prefix,
            data.NextContinuationToken,
            files
          )
        );
      } else {
        resolve(files);
      }
    });
  });
};


const calculateSize = async (bucket, prefix) => {
  let fileList = await getAllFileList(bucket, prefix);
  let size = 0;
  for (let i = 0; i < fileList.length; i++) {
    size += fileList[i].Size;
  }
  return size;
};

Now Just call calculateSize("YOUR_BUCKET_NAME","YOUR_FOLDER_NAME")