Why doesn't promise return from Lambda work while returning stream to handler for uploading to S3?

Question

I want to return a stream from a function for the main Lambda handler to create a pipe. This works:

const { S3Client } = require("@aws-sdk/client-s3")
const { Upload } = require('@aws-sdk/lib-storage')
const stream = require('stream')
const s3Region = 'us-east-1'
const bucketname = "my_bucket"

exports.handler = function (event, context, callback) {
    let streamfrom = stream.Readable.from(["four"])
    getS3Stream()
        .then(streamto => {
            stream.pipeline(
                streamfrom,
                streamto,
                () => {
                    callback(null, { 'statusCode': 200 })
                })
        })
}

function getS3Stream() {
    return new Promise(resolve => {
        const pass = new stream.PassThrough()
        const upload = new Upload({
            client: new S3Client({ region: s3Region }),
            params: {
                Bucket: bucketname,
                Key: "test/test.txt",
                Body: pass
            }
        })
        upload.done().then((res, error) => {
            if (error) { reject(error) }
            console.log("s3 uploaded")
        })
        resolve(pass)
    })
}

But I want the handler function to return a promise instead of using a callback, at which point it no longer works:

exports.handler = async function (event, context) {
    return new Promise(resolve => {
        let streamfrom = stream.Readable.from(["five5"])
        getS3Stream()
            .then(streamto => {
                stream.pipeline(
                    streamfrom,
                    streamto,
                    () => {
                        resolve({ 'statusCode': 200 })
                    })
            })
    })
}

It returns {"statusCode":200}, but "s3 uploaded" is not printed, and the file does not appear in S3. Am I misunderstanding something about how to use promises here?

`upload.done().then((res, error) => {` looks wrong. And notice that your `getS3Stream` function shouldn't return a promise at all, it just constructs a stream and can synchronously `return pass`. — Bergi, May 23 '22 at 13:45
I will look into that first one, maybe you are right. The promise returning is actually a requirement external to this code, because there are other types of streams that can be returned in the real program, which do require async. — Jon Wilson, May 23 '22 at 14:01
@Bergi The `upload.done()` function, for some reason, is required for it to work. It can even just be `upload.done().then(()=>{console.log('ok')});` but without it even the original version of my code does not work. If it was in an async function you could just do `await upload.done();` — Jon Wilson, May 23 '22 at 18:10
My guess would be that you *have* to wait for it, of course, you'd do that only *after* creating the pipeline, which doesn't really fit into your coding pattern. Does AWS document what this does? — Bergi, May 23 '22 at 18:33
@jarmod That won't make a difference. OP is not attempting to use `async`/`await`. — Bergi, May 23 '22 at 18:38
I haven't verified the behavior but the [Lambda docs](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html#nodejs-handler-async) indicate: "Functions must use the async keyword to use these methods (return or throw) to return a response or error." — jarmod, May 23 '22 at 18:45
@jarmod, if I made it an `async` function then I would use a `return` instead of `resolve` to return a Promise. I am using a Promise constructor (`new Promise()`) here mainly because I need to return from within a nested function: a `return` would not return to the outer function. If you know how to make that work, show me the code. — Jon Wilson, May 24 '22 at 11:20
Not sure I understand. You *are* returning a promise (that will later be resolved) in the final code segment shown in your post. That's entirely normal and the docs show exactly this case, except the exported function is decorated as async in the docs. I don't know that that actually makes a difference, per Bergi's comment, but I'm just alerting you to the specific recommendation in that doc. — jarmod, May 24 '22 at 12:14

Noel Llevares · Answer 1 · 2022-05-25T14:54:22.833

0

If you want your handler to return a Promise, you have to mark it as async.

I have not found any documentation on this but I believe the runtime caller first checks the handler definition so it know how it should call it.

Something like:

if (handler.constructor.name === 'AsyncFunction') {
   result = await handler(event, context)
   handleResult(null, result)
} else if (handler.constructor.name === 'Function') {
   handler(event, context, handleResult)
} else {}

So, going back to your original code, it can be simply

exports.handler = async function (event, context) {
  return new Promise(resolve => {
    let streamfrom = stream.Readable.from(["five5"])
    getS3Stream()
      .then(streamto => {
        stream.pipeline(
          streamfrom,
          streamto,
          () => resolve({'statusCode': 200})
        )
      })
  })
}

function getS3Stream() {
  return new Promise(resolve => {
    const pass = new stream.PassThrough()
    const upload = new Upload({
      client: new S3Client({region: s3Region}),
      params: {
        Bucket: bucketname,
        Key: "test/test.txt",
        Body: pass
      }
    })
    upload.done().then((res, error) => {
      if (error) {
        reject(error)
      }
      console.log("s3 uploaded")
      resolve(pass)
    })
  })
}

edited May 25 '22 at 14:54

answered May 25 '22 at 14:04

Noel Llevares

15,018
3
57
81

Were you able to validate this? I was not able to (yesterday). The symptoms were the same as the OP's report. – jarmod May 25 '22 at 14:15
I think you are correct, it should have the async, but it does not make it work in this case. In a Lambda, the async keyword, in addition to its normal features, also tells the runtime caller not to expect a callback, but that a Promise will be returned. I will edit the question. – Jon Wilson May 25 '22 at 14:41
Try moving the `resolve(pass)` to the callback that you pass to `upload.done().then()`. – Noel Llevares May 25 '22 at 14:55
That doesn't work, because what I am trying to do it return the stream itself, not the results of the stream. – Jon Wilson May 25 '22 at 15:20
"*I believe the runtime caller first checks the handler definition*" - while that might be the case (I don't know either), one should point out that this would be a bad practice. The standard approach to supporting both callbacks and promises is to call the function and simply check whether it returns a promise or not, or to check the function's `.length` if absolutely necessary – Bergi May 25 '22 at 20:45

score 0 · Answer 2 · answered May 26 '22 at 15:28

Inspired by this I came up with a solution that almost meets my qualifications. The function doesn't return just the stream, it also returns the promise. I am still hoping someone will come up with something that works without that crutch.

const { S3Client } = require("@aws-sdk/client-s3")
const { Upload } = require('@aws-sdk/lib-storage')

const stream = require('stream');
const s3Region = 'us-east-1'
const bucketname = "my_bucket"

exports.handler = async function (event, context) {
    return new Promise(resolve => {
        let streamfrom = stream.Readable.from(["five5"])
        getS3Stream()
        .then(({streamto, promise})=>{
            stream.pipeline(
                streamfrom,
                streamto,
                () => promise.then(()=>resolve({ 'statusCode': 200 }))        
            )
        })
    })
}

async function getS3Stream() {
    const streamto = new stream.PassThrough()
    const s3Client = new S3Client({ region: s3Region })
    const upload = new Upload({
        client: s3Client,
        params: {
            Bucket: bucketname,
            Key: "test/test.txt",
            Body: streamto
        }
    })
    let promise = upload.done()
    return({streamto,promise})
}

Why doesn't promise return from Lambda work while returning stream to handler for uploading to S3?

2 Answers2