Does Amazon S3 have a limit to MaxKeys when calling ListObjects?

Question

I always thought there was a 1,000 key limit when calling ListObjects in Amazon S3. However, I just made a call and it's pulling 1,080. But even their docs say there is a limit of 1,000.

I tried setting the MaxKeys setting to 1,000 but it still pulls 1,080 results. My code:

$iterator = $s3->getIterator('ListObjects', array(
    'Bucket' => 'BUCKETNAME',
    'MaxKeys' => 1000
));

It is however pulling folders as keys. But I certainly don't have 80 of them.

Two questions:

Is my code wrong?
Has Amazon lifted the 1000 key restriction? Is there a new limit?

score 13 · Accepted Answer · answered Aug 29 '13 at 14:17

13

The S3 API limit hasn't changed, it's still limited to a maximum of 1000 keys/response.

With the PHP SDK v1 a single request returned up to 1000 keys and to get the rest you needed to do a second request with the marker option.

The new PHP SDK (v2) has a concept of Iterators which abstracts the process of doing these multiple, consecutive requests. This makes getting ALL of your objects much easier.

answered Aug 29 '13 at 14:17

dcro

13,294
4
66
75

1

Ah, thank you so much again @dcro. Feeling a bit silly now... That's twice you've helped me today. I owe you a beer! But I'll give you a vote instead :) – Ben Sinclair Aug 29 '13 at 14:20
Yeah, iterators in SDK 2 simplify a lot of the work around fetching results across pages in a more memory-efficient way. – Ryan Parman Aug 31 '13 at 04:18

score 1 · Answer 2 · answered Jul 14 '20 at 06:16

By default the API returns up to 1,000 key names. The response might contain fewer keys but will never contain more. A better implementation would be use the newer ListObjectsV2 API:

 List<S3ObjectSummary> docList=new ArrayList<>();
    ListObjectsV2Request req = new ListObjectsV2Request().withBucketName(bucketName).withPrefix(folderFullPath);
    ListObjectsV2Result listing;
    do{
        listing=this.getAmazonS3Client().listObjectsV2(req);
        docList.addAll(listing.getObjectSummaries());
        String token = listing.getNextContinuationToken();
        req.setContinuationToken(token);
        LOG.info("Next Continuation Token for listing documents is :"+token);
    }while (listing.isTruncated());

score 0 · Answer 3 · answered Oct 28 '21 at 13:02

0

AWS s3 gives a maximum of 1000 files list in order to get more than 1000 count to use this approach

for javascript use this approach

https://stackoverflow.com/a/69754448/8239116

answered Oct 28 '21 at 13:02

Aurangzaib Rana

4,028
1
14
23

Does Amazon S3 have a limit to MaxKeys when calling ListObjects?

3 Answers3