2

Kindly excuse my knowledge with NodeJs, as I've just started with it. I've following lambda function which isn't fetching list of objects (more than 1000) in S3 and stuck in infinite loop, resulting lambda timimg out. Not sure what's wrong here

Code:

console.log('Loading');
const AWS = require('aws-sdk');
var request=true;
const awsOptions = {
    region: "us-east-1"
};
const s3 = new AWS.S3(awsOptions);
var list = [];
exports.handler = async (event, context, callback) => {
    
    const SrcBucket = event.Records[0].s3.bucket.name;
    const trigger_file = event.Records[0].s3.object.key;

    var bucketParams = {
        Bucket: SrcBucket,
        Prefix: 'Test/'
    };
    do
    {
        s3.listObjects(bucketParams, (err, data) => {
            if (err)
                console.log("Error", err);
            else
            {
                list.push(data.Contents);
                if (data.IsTruncated)
                    bucketParams.Marker = data.NextMarker;
                else
                    request = false;
            }
        });
    } while (request);

    callback(null, {
        listLen: list.length
    });
Aniruddha
  • 837
  • 3
  • 13
  • 37
  • Does this answer your question? [aws-sdk S3: best way to list all keys with listObjectsV2](https://stackoverflow.com/questions/42394429/aws-sdk-s3-best-way-to-list-all-keys-with-listobjectsv2) – Aritra Chakraborty Mar 08 '21 at 19:36
  • @AritraChakraborty Thanks for the input. However, I'd want to understand, why the above mentioned code isn't working? – Aniruddha Mar 09 '21 at 09:01
  • Oh! that's because you are returning the `listlen` before the listObjects callback actually ends. Your listObjects is asynchronous but u are calling the callback outside of that internal callback. – Aritra Chakraborty Mar 09 '21 at 09:04
  • @AritraChakraborty And what's the way to tackle this? Even if I remove last callback, handler function will anyway call it. Should I `await s3.listObjects` – Aniruddha Mar 09 '21 at 09:09
  • that's why i added that question cause it has callback and the promise solution. check the first answer of that question they are using same solution with recursive callback. u have to do the same. Otherwise check the 2nd answer it has promise solution which is neat and clean. another solution would be to promisify the listObjects function and using async/await or promise.then – Aritra Chakraborty Mar 09 '21 at 09:14
  • @AritraChakraborty I tried the code from your suggestion : https://stackoverflow.com/questions/42394429/aws-sdk-s3-best-way-to-list-all-keys-with-listobjectsv but when I do all `allKeys.lenght` I get 0 as response. Why is that? The bucket has many many files. – Aniruddha Mar 09 '21 at 11:00

0 Answers0