0

I am running Amazon S3 ls recursively over a folder inside an Amazon S3 bucket, when I use this command to count total no of objects aws s3 ls $bucket/$line --recursive | wc -l And when i compute the no of objects via aws console by selecting the folder and clicking on "Get Total Size" , I get lesser no of objects for some folders while for other it is same as calculated via aws s3 ls --recursive command.

Does Amazon S3 ls computing more objects incorrectly ? if yes then how to correct ?

smac2020
  • 9,637
  • 4
  • 24
  • 38
Deepak S
  • 1,544
  • 3
  • 15
  • 33
  • Can you provide an example of how big a difference it is? – Marcin May 23 '20 at 06:29
  • It might be including/excluding zero-length objects, which are used to force the appearance of a folder. You'll really need to find a specific folder that is mis-reporting and then closely examine the data being returned to identify the difference. What is your actual goal? Do you want to count all objects in a bucket, or just one folder, or all folders? If you need to do this regularly, you could instead use [Amazon S3 Inventory](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html), which can provide a daily/weekly CSV listing of all objects in the bucket. – John Rotenstein May 23 '20 at 06:52
  • difference is big - say : 3610016 vs 10733217 – Deepak S May 23 '20 at 07:34
  • @JohnRotenstein It is a one time activity , goal is to find total no of objects in a single folder of a bucket not all folders together, say one folder . – Deepak S May 23 '20 at 07:36
  • Similar method: [How can I tell how many objects I've stored in an S3 bucket?](https://stackoverflow.com/a/39111698/174777) – John Rotenstein May 23 '20 at 08:05
  • Would you want to include objects in a sub-folder of the one you are counting? – John Rotenstein May 23 '20 at 08:05
  • yes i need, so I am counting recursively, but i want to know why there is a difference between console count and one from aws s3 ls --recursive ? – Deepak S May 23 '20 at 08:47

0 Answers0