How do I make a form-data request in node pulling the files from s3

Question

Hey everyone so I am trying to make this type of request in nodejs. I assume you can do it with multer but there is one major catch I don't want to download the file or upload it from a form I want to pull it directly from s3, get the object and send it as a file along with the other data to my route. Is it possible to do that?

score 1 · Answer 1 · answered Aug 19 '19 at 14:16

1

Yes it's completely possible. Assuming you know your way around the aws-sdk, you can create a method for retrieving the file and use this method to get the data in your route and do whatever you please with them.

Example: (Helper Method)

    getDataFromS3(filename, bucket, callback) {

      var params = {
      Bucket: bucket, 
      Key: filename
     };

     s3.getObject(params, function(err, data) {
       if (err) {
           callback(true, err.stack); // an error occurred
       }
       else  {
          callback(false, data);     //success in retrieving data.
       }  
   });

 }

Your Route:

app.post('/something', (req, res) => {
    var s3Object = getDataFromS3('filename', 'bucket', (err, file) => {
        if(err) {
            return res.json({ message: 'File retrieval failed' });
         }

        var routeProperties = {};
        routeProperties.file = file;
        routeProperties.someOtherdata = req.body.someOtherData;

        return res.json({routeProperties});
    });
});

Of course, the code might not be totally correct. But this is an approach that you can use to get what you want. Hope this helps.

answered Aug 19 '19 at 14:16

Rohit Kashyap

1,553
1
10
16

can I put this in async await and how do I actually send it from the route once I get the data – codernoob7 Aug 19 '19 at 15:05
Yes, you can convert it to return a promise rather than a callback function. But that is something entirely else. After executing this function in the route, inside the function's callback, you can get the data as mentioned in the code. Return an error if there was an error while fetching else create an empty object, append your properties to that object and send the object as a response from the route using `res.json({ routeProperties })`. – Rohit Kashyap Aug 19 '19 at 15:17
Can you give me an example of how you would get the file and than send it via post route in the format shown in the picture of the postman route – codernoob7 Aug 19 '19 at 15:36
Wow, no. I have shown you the approach, the rest is up to you. This is the only way you can learn. Stackoverlow is a community of people who can show you a way and not write the whole code for you. – Rohit Kashyap Aug 19 '19 at 15:41

sdooo · Answer 2 · 2019-08-20T12:33:53.310

There are two ways that I see here, you can either:

pipe this request to user, it means that you still download it and pass it through but you don't save it anywhere, just stream it through your backend.

There is a very similar question asked here: Streaming file from S3 with Express including information on length and filetype

I'm just gonna copy & paste code snippet just for the reference how it could be done

function sendResponseStream(req, res){
  const s3 = new AWS.S3();
  s3.getObject({Bucket: myBucket, Key: myFile})
    .createReadStream()
    .pipe(res);
}

if the file gets too big for you to easily handle, create presigned URL in S3 and send it through. User then can download the file himself straight from S3 for a limited amount of time, more details here: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html

How do I make a form-data request in node pulling the files from s3

2 Answers2