Append items to an array from an async function

Question

I am having trouble appending items to an array and would like assistance in performing such. I have reviewed this and understood zero of it.

Here is my current code which is being ran in AWS Lambda (Node.js 10.x):

var sesData = ["array0", "array1"];

function onScan(err, data) {
        if (err) {
            console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
        } else {        
            console.log("Scan succeeded.");
            data.Items.forEach(function(itemdata) {
                // append new value to the sesData array
                sesData.push("Item :" + JSON.stringify(itemdata));
                console.log(sesData);
                console.log("Item :",JSON.stringify(itemdata));
            });

            // continue scanning if we have more items in case it is lots of data
            if (typeof data.LastEvaluatedKey != "undefined") {
                console.log("Scanning for more...");
                params2.ExclusiveStartKey = data.LastEvaluatedKey;
                dynamoDB.scan(params2, onScan);
            }
        }
    }

  function generateEmailParams (body) {
      return {
        Source: myEmail,
        Destination: { ToAddresses: [myEmail] },
        ReplyToAddresses: [myEmail],
        Message: {
          Body: {
            Text: {
              Charset: 'UTF-8',
              Data: `Message sent. \nContent: \n${sesData}`
            }
          },
          Subject: {
            Charset: 'UTF-8',
            Data: `Subject`
          }
        }
      }
    }
    //End email Params

exports.handler = function(event, context) {
    console.log("Incoming: ", event);
    dynamoDB.scan(params2, onScan); // scanning DDB
    console.log('===SENDING EMAIL===');
    const emailParams = generateEmailParams(event.body)
    var email = ses.sendEmail(emailParams, function(err, data){
        if(err) console.log(err);
        else {
            console.log("===EMAIL SENT===");
            console.log(data); // log data
            console.log("EMAIL CODE END"); //log end of email
            console.log('EMAIL: ', email); // log email
            context.succeed(event);

        }
    });

};

All the ses items are just sending the onScan function data via email. That works fine and is not the trouble, its that sesData never appends. The console.log’s print out the data from dynamoDB without issue but the push to array does not work.

After searching all over the internet, I do not really understand what’s going on as there are no errors so I am missing some sort of logic.

Answer 1

Async means the order of the code doesn't get executed in the same order you wrote it.

dynamoDB.Scan is a async function. You are talking to your DynamoDB, and it takes time, it might be only few ms, but NodeJS wil continue on to the next line of code while it finishes the scan function.

Lets take the following example

let count = 1
console.log(count)

setTimeout(function() {
  count = 2
  console.log(count)
}, 1000)

count = 3
console.log(count)

setTimeout is a async function, it executes after x ms, in this case 1000 ms = 1 sec. So its similar to your dynamoDB.scan function, it's starts right away, but it takes some time to finish and meanwhile, nodeJS will continue running your code line by line.

So the order the code is 1, 2, 3. But when you run the snippet it comes 1,3,2. Even if you set the timeout to 0 ms, it will still be 1,3,2

let count = 1
console.log(count)

setTimeout(function() {
  count = 2
  console.log(count)
}, 0)

count = 3
console.log(count)

This is because its an async function and will be put on the bottom of the callstack. Callstack is a fancy javascript word.

To understand this checkout: https://www.youtube.com/watch?v=8aGhZQkoFbQ Its a really good video that explains how JavaScript works and not to difficult to understand.

Note that the onScan function is a callback function and is executed when the dynamoDb.scan method has finished. So its like "Hey DynamoDB, call the scan function and then do this onScan stuff I created"

You programemd it so that when DynamoDB.scan has finished it should call onScan, which adds to sesData array, but the generateParams function is outside the callback so it gets called right way after you called dynamoDb.scan, not after it has finsed.

So what is happening in your code is this:

1. You create array sesData
2. You call dynamoDB.scan and pass it the onScan callback function.
3. The function starts, but it async and it takes time to connect to dynamoDB, etc
4. You call generateParams, but the onScan function hasn't finished.

Your code generated the email before the onScan function added items to sesData.

To fix this, you need to:

1. Include the generateParams in the onScan callback.
2. Use promise chains with .then
3. Use async/await

I haven't used AWS but quick googling shows that scan can return a promise by doing

dyanmoDb.scan(params).promise()

Notice that there is no callback function here, because it returns a promise.

You can then do something like

exports.handler = async function(event, context) {
    ...
    await dynamoDb.scan(params).promise()
    ...
    console.log(sesData)
}