Why is this lambda function not copying S3 objects?

Question

I've just created a lambda function with runtime Node.js 18.x. It retrieves a list of S3 objects from one bucket, but the copy is not working, even though the status returned is 'Successful'. I've also observed that the console.log before the copy command is output, but not the console.log following the command.

Does anyone see anything obvious that I've overlooked?

import { S3Client, ListObjectsV2Command, CopyObjectCommand } from "@aws-sdk/client-s3"

const client = new S3Client({})

export const handler = async (event) => {
  const sourceBucket = 'in-bucket'
  const destBucket = 'complete-bucket'
  const dateSuffix = '-' + (new Date()).toISOString().substring(0,10).replaceAll('-','')
  const commandList = new ListObjectsV2Command({Bucket: sourceBucket})

  try {
    let isTruncated = true

    while (isTruncated) {
      const { Contents, IsTruncated, NextContinuationToken } = await client.send(commandList)
      Contents.forEach(async (c) => {
        const fn = c.Key
        const options = {
          CopySource: encodeURI(`${sourceBucket}/${fn}`),
          Bucket: destBucket,
          Key: encodeURI(`${fn}${dateSuffix}`)
        }
        console.log(`moving ${fn}`)
        const commandCopy = new CopyObjectCommand(options)
        let response = await client.send(commandCopy)
        console.log(`response: ${response}`)
      })
      isTruncated = IsTruncated
      commandList.input.ContinuationToken = NextContinuationToken
    }
    console.log('Finished')

  } catch (err) {
    console.error(err)
  }
}

Execution results:

Test Event Name
(unsaved) test event

Response
null

Function Logs
START RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2 Version: $LATEST
2023-07-27T17:50:28.279Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file01.csv
2023-07-27T17:50:28.281Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file02.csv
2023-07-27T17:50:28.337Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file03.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file04.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file05.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file06.csv
2023-07-27T17:50:28.338Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file07.csv
2023-07-27T17:50:28.339Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file08.csv
2023-07-27T17:50:28.339Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file09.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file10.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file11.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file12.csv
2023-07-27T17:50:28.340Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file13.csv
2023-07-27T17:50:28.341Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file14.csv
2023-07-27T17:50:28.341Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file15.csv
2023-07-27T17:50:28.397Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    moving file16.csv
2023-07-27T17:50:28.397Z    dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2    INFO    Finished
END RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2
REPORT RequestId: dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2  Duration: 1178.43 ms    Billed Duration: 1179 ms    Memory Size: 128 MB Max Memory Used: 103 MB Init Duration: 621.73 ms

Request ID
dbb33900-00e4-4e2f-8afe-9e9dd4c00cc2

I applied sample code, e.g., https://docs.aws.amazon.com/AmazonS3/latest/userguide/example_s3_CopyObject_section.html Since the first command returned all the expected key values, I expected the second command to copy the objects from the source bucket to the destination bucket.

You have a `forEach` with an async function. This most likely wont work as you might expect, the `forEach` loop, and hence your lambda function, will most likely finish before any copying is happening. See this Stackoverflow answer regarding usage of async function in `forEach`: https://stackoverflow.com/a/70946414/7661119 — Ervin Szilagyi, Jul 27 '23 at 18:56
Thank you! That was exactly the issue. When I replaced Contents.forEach(async (c) => { with for (const c of Contents) { it worked as expected. — Ed Trembicki-Guy, Jul 27 '23 at 19:50

score 1 · Answer 1 · edited Jul 27 '23 at 22:45

Thank you, Ervin Szilagyi, for pointing out:

You have a forEach with an async function. This most likely wont work as you might expect, the forEach loop, and hence your lambda function, will most likely finish before any copying is happening. See this Stackoverflow answer regarding usage of async function in forEach: stackoverflow.com/a/70946414/766111

That was exactly the issue. When I replaced:

Contents.forEach(async (c) => {

with:

for (const c of Contents) {

it worked as expected.

Why is this lambda function not copying S3 objects?

1 Answers1