GET list of objects located under a specific S3 folder

Question

I am trying to GET a list of objects located under a specific folder in an S3 bucket using a query-string which takes the foldername as the parameter and list all objects which match that specific folder using Node JS aws-sdk

For example: http://localhost:3000/listobjects?foldername=xxx

Please suggest how to implement this functionality.

score 51 · Answer 1 · answered May 16 '18 at 09:14

You can specify the prefix while calling the getObject or listObjectsV2 in aws-sdk

var params = {
  Bucket: 'STRING_VALUE', /* required */
  Prefix: 'STRING_VALUE'  // Can be your folder name
};
s3.listObjectsV2(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log(data);           // successful response
});

By the way, S3 doesn't have folders. It is just a prefix. It shows you the folder structure to make it easy for you too navigate and see files.

Source: AWS SDK Javascript

score 26 · Answer 2 · edited Aug 01 '22 at 13:18

26

You forget to mention folder into s3 bucket, anyways this code works for me

var params = {
  Bucket: 'Bucket_Name',
  Delimiter: '/',
  Prefix: 'foldername/'
};

s3Bucket.listObjects(params, function(err, data) {
  if (err) {
    return 'There was an error viewing your album: ' + err.message
  } else {
    console.log(data.Contents,"<<<all content");
                
    data.Contents.forEach(function(obj,index) {
      console.log(obj.Key,"<<<file path")
    })
  }
})

edited Aug 01 '22 at 13:18

Tyler2P

2,324
26
22
31

answered Aug 28 '18 at 17:30

Amit Shakya

962
3
16
30

3

Works like a charm! It is important to note in the above example that Prefix: 'foldername/' has '/' at the end. Otherwise it does not work. – v.j Feb 07 '19 at 06:47
This lists in my case all files in the folder but in additon the folder itself too. Looks like there folder is always listed first – Tobi Apr 26 '19 at 08:33

score 14 · Answer 3 · answered Apr 26 '19 at 08:36

14

Starting with index = 1 in the loop excludes the folder itself + just lists the files in the folder:

const s3 = new AWS.S3();

const params = {
    Bucket: bucketname,
    Delimiter: '/',
    Prefix: s3Folder + '/'
};

const data = await s3.listObjects(params).promise();

for (let index = 1; index < data['Contents'].length; index++) {
    console.log(data['Contents'][index]['Key'])        
}

answered Apr 26 '19 at 08:36

Tobi

1,702
2
23
42

4

Instead of depending on `index = 1` to skip the folder key, you can use `StartAfter: s3Folder + '/'` in the params object. StartAfter sets the starting S3 key value, allowing the loop to iterate from the standard `0`. https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property – Shawn Moore Sep 24 '19 at 22:33

score 6 · Answer 4 · edited Dec 17 '21 at 20:50

AWS s3 gives a maximum of 1000 files list in order to get more than 1000 count use this approach

export function getListingS3(prefix) {
  return new Promise((resolve, reject) => {
    try {
      let params = {
        Bucket: AWS_S3.BUCKET_NAME,
        MaxKeys: 1000,
        Prefix: prefix,
        Delimiter: prefix
      };
      const allKeys = [];
      listAllKeys();
      function listAllKeys() {
        s3.listObjectsV2(params, function (err, data) {
          if (err) {
            reject(err)
          } else {
            var contents = data.Contents;
            contents.forEach(function (content) {
              allKeys.push(content.Key);
            });

            if (data.IsTruncated) {
              params.ContinuationToken = data.NextContinuationToken;
              console.log("get further list...");
              listAllKeys();
            } else {
              resolve(allKeys);
            }
          }
        });
      }
    } catch (e) {
      reject(e);
    }

  });
}```

With a few adaptations, worked like a charm using the ContinuationToken property. — rscherer, Nov 09 '21 at 20:10

GET list of objects located under a specific S3 folder

4 Answers4

Linked