0

In my S3 buckets there are many directories at first level. Each containing data of it's own.

- Bucket
   -- dir_1
       -- file1
        -- file2
   -- dir_2
       -- somefile

I want to fetch total size (including all the subobjects under it) of each first level object (dir_1, dir_2) in the bucket. I tried both ListObjects and ListObjectsV2 with different Prefix parameter values but either it's giving no data or all the data including object within objects

Saqib
  • 2,470
  • 3
  • 19
  • 32
  • What do you mean by "fetch size of each first level object"? Do you mean you would like to know the total storage space taken by all files within `dir_1`? Or are you simply wanting to see the size of individual objects in the top-most directory? – John Rotenstein Aug 05 '20 at 10:29
  • @JohnRotenstein I want to get total size of dir_1 including all of it's sub objects – Saqib Aug 06 '20 at 12:22
  • Directories do not actually exist in Amazon S3, so I shall interpret your requirements as "calculate the total size of objects with the Prefix of `dir_1/`. You will need to calculate this yourself, as explained by @Marcin below. – John Rotenstein Aug 06 '20 at 12:24

1 Answers1

2

Sadly, there is no API call that can give you size of your folders/prefixes.

Normally for that you have too iterate over all the objects yourself, get their data and calculate the size.

Since this is a common request, there are many recipes already made for that. For example, here or here.

The alternative is to use Amazon S3 inventory. The inventory would produce a csv file daily with the metadata of your objects. Then you could parse it to calculate the sizes based on the prefixes.

Marcin
  • 215,873
  • 14
  • 235
  • 294