6

I am trying to find a faster way to count all of the objects within an s3 bucket using Amazon's AWS SDK.

private static int getBucketFileCount(AmazonS3 s3, ListObjectsV2Request req) {
   ListObjectsV2Result result;
   int fileCount = 0;
   log.info("Counting s3 files");

   do {
      result = s3.listObjectsV2(req);
      for (S3ObjectSummary objectSummary : result.getObjectSummaries()) {
         fileCount++;
      }
      req.setContinuationToken(result.getNextContinuationToken());

   } while (result.isTruncated() == true);
       return fileCount;
}

However, this method is very slow and I have not been able to figure out a way to do it properly. I have found another answer that sort of helps, but cannot figure out the implementation exactly. Will getObjectSummaries get the count of objects stored in a S3 Bucket?

How do I use the getNextMarker() function with my current implementation? What do I need to change?

Jordan
  • 75
  • 1
  • 5
  • I noticed you have `fileCount += result.getKeyCount()` as well as `fileCount++` inside a for loop. Are you double counting? – kevdoran Aug 23 '17 at 21:30
  • Ah my bad, that was something I forgot to remove – Jordan Aug 23 '17 at 21:36
  • If you don't require an up-to-date count value, [Amazon S3 Inventory](http://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html) is a scheduled alternative to the Amazon S3 synchronous List API operation. It provides a comma-separated values (.csv) flat-file output of your objects and their corresponding metadata on a daily or weekly basis. – Khalid T. Aug 24 '17 at 06:44
  • refer https://stackoverflow.com/questions/28113605/will-getobjectsummaries-get-the-count-of-objects-stored-in-a-s3-bucket for answer. – Amit Aug 24 '17 at 10:44
  • @AmitKhandelwal I already linked to that post. I still do not understand how exactly to use those functions since I am not well versed in aws sdk – Jordan Aug 24 '17 at 18:37
  • Does this answer your question? [Will getObjectSummaries get the count of objects stored in a S3 Bucket?](https://stackoverflow.com/questions/28113605/will-getobjectsummaries-get-the-count-of-objects-stored-in-a-s3-bucket) – Farid Nouri Neshat Dec 12 '21 at 17:07

0 Answers0