aws lambda how to store an image retrieved via https in S3

Question

I am trying to write a lambda script that can pull an image from a site and store it in S3. The problem I'm having is what kind of object to pass as the Body attribute into the S3.putObject method. In the documentation here it says it should be either new Buffer('...') || 'STRING_VALUE' || streamObject, but I'm not sure how to convert the https response into one of those. Here is what I've tried:

var AWS = require('aws-sdk');
var https = require('https');
var Readable = require('stream').Readable;
var s3 = new AWS.S3();
var fs = require('fs');

var url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/500px-AmazonWebservices_Logo.svg.png';

exports.handler = function(event, context) {
  https.get(url, function(response) {
    var params = {
     Bucket: 'example',
     Key: 'aws-logo.png',
     Body: response  // fs.createReadStream(response); doesn't work, arg should be a path to a file...
                     // just putting response errors out with "Cannot determine length of [object Object]"
    };

    s3.putObject(params, function(err, data) {
      if (err) {
        console.error(err, err.stack);
      } else {
        console.log(data);
      }
    });
  });
};

Please refer to http://stackoverflow.com/questions/19016130/pushing-binary-data-to-amazon-s3-using-node-js — Shibashis, May 17 '16 at 02:52
If I'm not mistaken, it looks like in that example the script is being run on a local machine, they are using curl to to save the file to disk, and then creating a stream with the path to that saved file. Since I'm on lambda I can't save the file to disk — kleaver, May 17 '16 at 03:18
"Since I'm on lambda I can't save the file to disk". That's not correct. You have disk space available on Lambda to temporarily store files. — Mark B, May 17 '16 at 13:06

Alexis N-o · Accepted Answer · 2018-01-09T16:32:50.573

As indicated in the comments, Lambda allows to save files in /tmp. But you don't really need it...

response does not contain the content of the file, but the http response (with its status code and headers).

You could try something like this:

var AWS = require('aws-sdk');
var https = require('https');
var s3 = new AWS.S3();

var url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/AmazonWebservices_Logo.svg/500px-AmazonWebservices_Logo.svg.png';

exports.handler = function(event, context) {
  https.get(url, function(res) {
    var body = '';
    res.on('data', function(chunk) {
      // Agregates chunks
      body += chunk;
    });
    res.on('end', function() {
      // Once you received all chunks, send to S3
      var params = {
        Bucket: 'example',
        Key: 'aws-logo.png',
        Body: body
      };
      s3.putObject(params, function(err, data) {
        if (err) {
          console.error(err, err.stack);
        } else {
          console.log(data);
        }
      });
    });
  });
};

just for the record "s3" is undefined here, I had to add: ```var s3 = new AWS.S3();``` — Eli Albért, Jan 09 '18 at 16:18
Any idea what s3 resources (apart from storage space) would it consume to transfer the image to s3. How many put etc and any other charges it would incur? — Asad Feroz Ali, Apr 12 '18 at 08:06

score 3 · Answer 2 · answered May 17 '16 at 06:22

3

try this package https://www.npmjs.com/package/request

var request = require('request');

exports.handler = function (event, context) {
   s3.putObject({
       Bucket: 'example',
       Key: 'aws-logo.png',
       Body: request.get(url, {followRedirect: false})
   }, function (err, data) {
       if (err)  console.error(err, err.stack);
       else console.log(data);
   })
}

answered May 17 '16 at 06:22

KibGzr

2,053
14
15

This would work fine, but the `request` module isn't in core node so I chose @alexis-n-o answer because it didn't require loading that extra module – kleaver May 17 '16 at 15:19

aws lambda how to store an image retrieved via https in S3

2 Answers2