nodejs Lambda with S3 upload via API Gateway

Question

I've been trying to get a simple serverless API Gateway -> NodeJS Lambda -> S3 working however it appears that the Lambda just uploads corrupt files.

This code would download the file from a URL then straight upload to S3. I've tried both putObject & upload (with the different params) with no success. Looking at the file sizes when I download the original is is 24KB and the downloaded (corrupt) image from S3 is 44KB.

I simply test the application by doing a POST to the API Gateway URL.

Any ideas?

var url =
  "https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/500px-AmazonWebservices_Logo.svg.png"
module.exports.upload = function(event, context, callback) {
  https.get(url, function(res) {
    var body = ""
    res.on("data", function(chunk) {
      // Agregates chunks
      body += chunk
    })
    res.on("end", function() {
      console.log(body)

      // Once you received all chunks, send to S3 - putObject only
      var params = {
        Bucket: S3_BUCKET_NAME,
        Key: "aws-logo.png",
        Body: body
      }

      var s3Params = {
        Bucket: S3_BUCKET_NAME,
        Key: "aws-logo-upload.png",
        Body: body,
        ContentType: "image/png"
      }

      s3.upload(s3Params, function(err, data) {
        // s3.putObject(params, function(err, data) {
        if (err) {
          console.log("error")
          console.error(err, err.stack)
          callback(null, { statusCode: 404, error })
        } else {
          console.log("ok")
          console.log(data)
          let response = {
            statusCode: 200
          }
          callback(null, response)
        }
      })
    })
  })
}

Probably the the request execute twice. did you bind your lambda function to any events ? — Abderrahim Soubai-Elidrisi, Sep 09 '18 at 21:48
It is tied via my serverless.yml: functions: upload: handler: handler.upload events: - http: path: event/upload method: post — jamiem, Sep 09 '18 at 21:53
It does not work when I try to manually test/run it in the AWS Console. — jamiem, Sep 09 '18 at 22:04
Related: https://stackoverflow.com/questions/17836438/getting-binary-content-in-node-js-with-http-request — jarmod, Sep 09 '18 at 22:05
I did try with different binary encoding settings - along with when I was using API gateway to actually trigger it with image data in the HTTP Body with the same problem (this is my simplified code to locate the issue) — jamiem, Sep 09 '18 at 22:15

jarmod · Accepted Answer · 2018-09-10T00:23:11.380

The following code works for me outside of API Gateway/Lambda. It yields a PNG in S3 that's downloadable as a valid 23.7 KB image. I'd expect the equivalent to work in Lambda.

const AWS = require('aws-sdk');
const https = require('https');
const s3 = new AWS.S3();

const logourl =
  'https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/500px-AmazonWebservices_Logo.svg.png';

const getThenUpload = (url, callback) => {
  https.get(url, (res) => {
    const data = [];

    res.on('data', (chunk) => {
      data.push(chunk);
    });

    res.on('end', () => {
      const params = {
        Bucket: S3_BUCKET_NAME,
        Key: 'aws-logo-upload.png',
        Body: Buffer.concat(data),
        ContentType: 'image/png',
      };

      s3.upload(params, (err, rsp) => {
        if (err) {
          console.error(err, err.stack);
          callback(err, { statusCode: 404, err });
        } else {
          console.log(rsp);
          callback(null, { statusCode: 200 });
        }
      });
    });
  });
};

getThenUpload(logourl, (err, data) => {
  if (err) {
    console.error(`Error: ${err}`);
  } else {
    console.log(`Data: ${JSON.stringify(data)}`);
  }
});

What are you POSTing via API Gateway? The actual file contents, a URL pointing to the file, or something else? — jarmod, Sep 10 '18 at 12:46
Thanks for this :) worked! Had to change some of my API Gateway integration's. That way I could pass the raw data as Base64. — jamiem, Sep 11 '18 at 20:25

nodejs Lambda with S3 upload via API Gateway

1 Answers1