Stream File Directly to s3 using NodeJS+Express, aws-sdk

Question

I want to upload some large files directly to s3 via the browser with NodeJS, it is unclear how to prepare this file for upload to s3. There might be a better module (like Knox) to handle this case but I am not sure. Any thoughts?

File Object

  file: { 
     webkitRelativePath: '',
     lastModifiedDate: '2013-06-22T02:43:54.000Z',
     name: '04-Bro Safari & UFO! - Animal.mp3',
     type: 'audio/mp3',
     size: 11082039 
  }

S3 putObject

var params = {Bucket: 'bucket_name/'+req.user._id+'/folder', Key: req.body['file']['name'], Body: ???};
s3.putObject(params, function(err, data) {
    if (err)
      console.log(err);
    else
      console.log("Successfully uploaded data to myBucket/myKey");
});

Johann Philipp Strathausen · Accepted Answer · 2017-03-20T14:04:35.283

19

Streaming is now supported (see docs), simply pass the stream as the Body:

var fs = require('fs');
var someDataStream = fs.createReadStream('bigfile');
var s3 = new AWS.S3({ params: { Bucket: 'myBucket', Key: 'myKey' } });
s3.putObject({ Body: someDataStream, ... }, function(err, data) {
  // handle response
})

edited Mar 20 '17 at 14:04

answered Apr 04 '16 at 12:23

Johann Philipp Strathausen

5,706
3
31
39

5

Does this not require the file to be stored on hard disk before being uploaded to S3? I assumed OP wanted to skip the step of saving to disk and upload directly to S3 ... then again, I am a complete novice when it comes to file uploading. – AdamInTheOculus Mar 17 '17 at 14:24
You can use any stream for the upload - it doesn't have to come from the disc. – Johann Philipp Strathausen Mar 20 '17 at 14:05
this method is able to upload only smaller files. For better performance and control I am using `s3.upload()` – Ishank Aug 15 '18 at 06:07
1

Note that if you are piping this from another stream, you will receive an unknown content length error. I recommend using `upload` which does not have this problem and does have other advantages: https://stackoverflow.com/a/38442712/1277350 – LordParsley May 23 '19 at 14:22

score 3 · Answer 2 · answered Nov 04 '13 at 04:36

3

The s3.putObject() method does not stream, and from what I see, the s3 module doesn't support streaming. However, with Knox, you can use Client.putStream(). Using the file object from your question, you can do something like this:

var fs = require('fs');
var knox = require('knox');

var stream = fs.createReadStream('./file');
var client = knox.createClient({
  key: '<api-key-here>',
  secret: '<secret-here>',
  bucket: 'learnboost'
});

var headers = {
  'Content-Length': file.size,
  'Content-Type': file.type
};

client.putStream(stream, '/path.ext', headers, function(err, res) {
  // error or successful upload
});

answered Nov 04 '13 at 04:36

hexacyanide

88,222
31
159
162

thank you, I know my question was a bit vague as there are a lot of factors contributing - I had a feeling Knox would be what I need to use, thank you – Jeff Voss Nov 04 '13 at 04:58
So just to get it straight, this will work without uploading the file to the web server first? I want to avoid that entirely – Jeff Voss Nov 04 '13 at 05:05
This will stream the file to the server. The main difference between this and `putObject()` is that the file doesn't have to be entirely loaded into memory, therefore allowing you to upload extremely large files. – hexacyanide Nov 04 '13 at 05:07
I guess i'm only confused about the var stream = fs.createReadStream('./file'); doesn't this need to point to a file on the server? – Jeff Voss Nov 04 '13 at 05:10
The read stream points to a local file. It is the file that you intend to stream to the server. – hexacyanide Nov 04 '13 at 05:11
Yes so where would this file live, if I am attempting to upload directly from browser – Jeff Voss Nov 04 '13 at 05:26
To get a file to s3, you'd first need to upload it to the Node.js server, then the Node server would stream it to the s3 server. Since you tagged Express, the file would live wherever the `bodyParser()` middleware put it. You'd then have to check the `req.files` object to find where the client uploaded the file. Once you have the location, just create a read stream to that location, and stream it to s3. – hexacyanide Nov 04 '13 at 05:29
let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/40482/discussion-between-whiteb0x-and-hexacyanide) – Jeff Voss Nov 04 '13 at 05:32
Hey hexacyanide can you email me when you get a chance iamwhitebox@gmail.com – Jeff Voss Nov 06 '13 at 00:18
Why it doesn't work for big files? I try it for file of size > 990 MB and it gives me the following error: { [Error: write ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'write' } – Or Smith Aug 17 '14 at 13:46

score 1 · Answer 3 · edited May 23 '17 at 12:09

1

One option is to use multer-s3 instead: https://www.npmjs.com/package/multer-s3.

This post has some details also: Uploading images to S3 using NodeJS and Multer. How to upload whole file onFileUploadComplete

edited May 23 '17 at 12:09

Community

1
1

answered Jul 28 '15 at 01:39

Ankur Sanghi

266
3
14

score 0 · Answer 4 · answered Nov 04 '13 at 04:37

Your code isn't streaming. You need to see a call to pipe somewhere or at least code to pipe by hand by using data event handlers. You are probably using the express bodyParser middleware, which is NOT a streaming implementation. It stores the entire request body as a temporary file on the local filesystem.

I'm not going to provide specific suggestions because of the promising results I got from a web search for "node.js s3 stream". Spend 5 minutes reading, then post a snippet that is at least an attempt at streaming and we can help you get it right once you have something in the ballpark.

score 0 · Answer 5 · answered Aug 25 '21 at 17:19

The v3, the PutObjectCommand can not write file stream to S3. We need to use the @aws-sdk/lib-storage library for uploading buffers and streams.

Example:

const upload = async (fileStream) => {
    const uploadParams = {
        Bucket    : 'test-bucket',
        Key    : 'image1.png',
        Body: fileStream,
    }

    try {
        const parallelUpload = new Upload({
            client: s3Client,
            params: uploadParams,
        });

        console.log('Report progress..')
        parallelUpload.on("httpUploadProgress", (progress) => {
            console.log(progress);
        });

        await parallelUpload.done();
    } catch (e) {
        console.log(e);
    }
}

Ref - https://github.com/aws/aws-sdk-js-v3/blob/main/UPGRADING.md#s3-multipart-upload

Stream File Directly to s3 using NodeJS+Express, aws-sdk

5 Answers5

Linked