AWS S3: how do I see how much disk space is using

Question

I have AWS account. I'm using S3 to store backups from different servers. The question is there any information in the AWS console about how much disk space is in use in my S3 cloud?

Answer 1

The command line tool gives a nice summary by running:

aws s3 ls s3://mybucket --recursive --human-readable --summarize

Answer 2

Yippe - an update to AWS CLI allows you to recursively ls through buckets...

aws s3 ls s3://<bucketname> --recursive  | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'

Answer 3

To find out size of S3 bucket using AWS Console:

1. Click the S3 bucket name
2. Select "Metrics" tab
3. You should see "Bucket metrics" which by default includes "Total bucket size"

Answer 4

s3cmd can show you this by running s3cmd du, optionally passing the bucket name as an argument.

Answer 5

The AWS CLI now supports the --query parameter which takes a JMESPath expressions.

This means you can sum the size values given by list-objects using sum(Contents[].Size) and count like length(Contents[]).

This can be be run using the official AWS CLI as below and was introduced in Feb 2014

 aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Answer 6

On linux box that have python (with pip installer), grep and awk, install AWS CLI (command line tools for EC2, S3 and many other services)

sudo pip install awscli

then create a .awssecret file in your home folder with content as below (adjust key, secret and region as needed):

[default]
aws_access_key_id=<YOUR_KEY_HERE>
aws_secret_access_key=<YOUR_SECRET_KEY_HERE>
region=<AWS_REGION>

Make this file read-write to your user only:

sudo chmod 600 .awssecret

and export it to your environment

 export AWS_CONFIG_FILE=/home/<your_name>/.awssecret

then run in the terminal (this is a single line command, separated by \ for easy reading here):

aws s3 ls s3://<bucket_name>/foo/bar | \
grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | \
awk 'BEGIN {total=0}{total+=$3}END{print total/1024/1024" MB"}'

• the aws part lists the bucket (or optionally a 'sub-folder')
• the grep part removes (using -v) the lines that match the Regular Expression (using -E). ^$ is for blank line, -- is for the separator lines in the output of aws s3 ls
• the last awk simply add to total the 3rd colum of the resulting output (the size in KB) then display it at the end

NOTE this command works for the current bucket or 'folder', not recursively

Answer 7

Cloud watch also allows you to create metrics for your S3 bucket. It shows you metrics by the size and object count. Services> Management Tools> Cloud watch. Pick the region where your S3 bucket is and the size and object count metrics would be among those available metrics.

Answer 8

In addition to Christopher's answer.

If you need to count total size of versioned bucket use:

aws s3api list-object-versions --bucket BUCKETNAME --output json --query "[sum(Versions[].Size)]"

It counts both Latest and Archived versions.

Answer 9

Getting large buckets size via API (either aws cli or s4cmd) is quite slow. Here's my HowTo explaining how to parse S3 Usage Report using bash one liner:

cat report.csv | awk -F, '{printf "%.2f GB %s %s \n", $7/(1024**3 )/24, $4, $2}' | sort -n

Answer 10

Based on @cudds's answer:

function s3size()
{
    for path in $*; do
        size=$(aws s3 ls "s3://$path" --recursive | grep -v -E "(Bucket: |Prefix: |LastWriteTime|^$|--)" | awk 'BEGIN {total=0}{total+=$3}END{printf "%.2fGb\n", (total/1024/1024/1024)}')
        echo "[s3://$path]=[$size]"
    done
}

...

$ s3size bucket-a bucket-b/dir
[s3://bucket-a]=[24.04Gb]
[s3://bucket-b/dir]=[26.69Gb]

Also, Cyberduck conveniently allows for calculation of size for a bucket or a folder.

Answer 11

This is an old inquiry, but since I was looking for the answer I ran across it. Some of the answers made me remember I use S3 Browser to manage data. You can click on a bucket and hit properties and it shows you the total. Pretty simple. I highly recommend the browser: https://s3browser.com/default.aspx?v=6-1-1&fam=x64

Answer 12

You asked: information in AWS console about how much disk space is using on my S3 cloud?

I so to the Billing Dashboard and check the S3 usage in the current bill.

They give you the information - MTD - in Gb to 6 decimal points, IOW, to the Kb level.

It's broken down by region, but adding them up (assuming you use more than one region) is easy enough.

^{BTW: You may need specific IAM permissions to get to the Billing information.}

Answer 13

The AWS console wont show you this but you can use Bucket Explorer or Cloudberry Explorer to get the total size of a bucket. Both have free versions available.

Note: these products still have to get the size of each individual object, so it could take a long time for buckets with lots of objects.

Answer 14

Mini John's answer totally worked for me! Awesome... had to add

--region eu-west-1 

from Europe though

Answer 15

Well, you can do it also through an S3 client if you prefer a human friendly UI.

I use CrossFTP, which is free and cross-platform, and there you can right-click on the folder directory -> select "Properties..." -> click on "Calculate" button next to Size and voila.

Answer 16

s3admin is an opensource app (UI) that lets you browse buckets, calculate total size, show largest/smallest files. It's tailored for having a quick overview of your Buckets and their usage.

Answer 17

So I am going to add Storage Lens from AWS on here with the default dashboard.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-lens-optimize-storage.html?icmpid=docs_s3_hp_storage_lens_dashboards

It is really super useful for identify hidden cost of storage like "incomplete multipart uploads"

It really should probably be now the first port of call for answering this question before you now reach for the code.

Answer 18

Depending on how accurate you want your results to be, you can use AWS console, AWS CLI or AWS S3 storage lens to find out the total size of a bucket or how much space it is using.

Way 1: Using Console

Login to AWS Management
Open S3
Click on the “Matrics” tab
The total bucket size matrics show the size of your bucket

Note: Please note that this data is updated every 24 hours so the latest changes will not be there. But the benefit is that it's free to use and you are not charged for any S3 operation.

Here is how it looks like : Total Bucket Size

Using AWS CLI

aws s3 ls s3://bucket-name --recursive --human-readable --summarize

As expected, the CLI is running ls command so It will cost you money. So use as per your requirement.

If you want to check the storage lens way as well. you can check my post.

Answer 19

I use Cloud Turtle to get the size of individual buckets. If the bucket size exceeds >100 Gb, then it would take some time to display the size. Cloud turtle is freeware.