0

I'm using node-s3-client and want to save all contents of my bucket into an array. The problem is there is more than 1000 files in the bucket and this is the max. I've seen an example here I am just unsure how to use the Marker and isTruncated. I have the following code to get the first 1000 files

var s3 = require('s3');
var client = s3.createClient({
   maxAsyncS3: 20,
   s3RetryCount: 3,
   s3RetryDelay: 1000,
   multipartUploadThreshold: 20971520,
   multipartUploadSize: 15728640,
   s3Options: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID,
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
   }
});

var params = {
    Bucket: process.env.AWS_XML_BUCKET
}
client.s3.listObjects(params, function(err, data) {
    if(err)throw err;
    console.log(data.Contents);
});

If anyone knows how I could implement something like the example listed above to list all keys that would be great. As I mentioned when I posted my question I have seen that example and linked it. I am just unsure how to use Marker and get it working.

Community
  • 1
  • 1
nasoj1100
  • 528
  • 9
  • 15
  • This question is actually a duplicate of this: http://stackoverflow.com/questions/9437581/node-js-amazon-s3-how-to-iterate-through-all-files-in-a-bucket which was the top result in Google when I searched for "nodejs s3 marker truncated". The highest voted answer to that question gives the code to do exactly what you are asking for. – Mark B Feb 23 '16 at 15:26
  • yeah that is what i linked I am just not sure how to use Marker as I mentioned – nasoj1100 Feb 23 '16 at 15:39
  • Right, as as I mentioned, you can look at the top voted answer to that question and see exactly how to use it. You could literally copy/paste that top voted answer into your code and have your solution. – Mark B Feb 23 '16 at 15:47
  • what do I pass in as marker then when I call listAllKeys – nasoj1100 Feb 23 '16 at 15:49
  • Yeah I apologize, that answer requires you to start with a Marker value, which doesn't make sense. Look at the answer on that question provided by "Ken Lin", he corrects that issue in his answer. – Mark B Feb 23 '16 at 15:56

1 Answers1

0

I came up with a temporary solution. I just created an array of the folder names and made a call to each one separately. As none of the folders have over 1000 files, this will do for now.

nasoj1100
  • 528
  • 9
  • 15