Upload a binary file to S3 using AWS SDK for Node.js

Question

Update: For future reference, Amazon have now updated the documentation from what was there at time of asking. As per @Loren Segal's comment below:-

We've corrected the docs in the latest preview release to document this parameter properly. Sorry about the mixup!

I'm trying out the developer preview of the AWS SDK for Node.Js and want to upload a zipped tarball to S3 using putObject.

According to the documentation, the Body parameter should be...

Body - (Base64 Encoded Data)

...therefore, I'm trying out the following code...

var AWS = require('aws-sdk'),
    fs = require('fs');

// For dev purposes only
AWS.config.update({ accessKeyId: 'key', secretAccessKey: 'secret' });

// Read in the file, convert it to base64, store to S3
fs.readFile('myarchive.tgz', function (err, data) {
  if (err) { throw err; }

  var base64data = new Buffer(data, 'binary').toString('base64');

  var s3 = new AWS.S3();
  s3.client.putObject({
    Bucket: 'mybucketname',
    Key: 'myarchive.tgz',
    Body: base64data
  }).done(function (resp) {
    console.log('Successfully uploaded package.');
  });

});

Whilst I can then see the file in S3, if I download it and attempt to decompress it I get an error that the file is corrupted. Therefore it seems that my method for 'base64 encoded data' is off.

Can someone please help me to upload a binary file using putObject?

score 65 · Accepted Answer · answered Dec 10 '12 at 22:17

65

You don't need to convert the buffer to a base64 string. Just set body to data and it will work.

answered Dec 10 '12 at 22:17

AndyD

5,252
35
32

1

That seems to be it! Not sure the reasoning for the 'base64' mention in the docs then – isNaN1247 Dec 10 '12 at 22:22
11

We've corrected the docs in the latest preview release to document this parameter properly. Sorry about the mixup! – Loren Segal Dec 13 '12 at 00:46
3

Is it possible to pass one Stream instead of data ? For instance, i wanna sent one file that have 50MB. Could I pass a readable stream and the S3.client pipe internally to the S3 ? – CHAPa Apr 28 '13 at 11:24
1

Yes that's possible. see here: http://stackoverflow.com/questions/15817746/stream-uploading-an-gm-resized-image-to-s3-with-aws-sdk – AndyD Apr 28 '13 at 18:20

CaptEmulation · Answer 2 · 2014-11-27T00:27:09.633

27

Here is a way to send a file using streams, which might be necessary for large files and will generally reduce memory overhead:

var AWS = require('aws-sdk'),
    fs = require('fs');

// For dev purposes only
AWS.config.update({ accessKeyId: 'key', secretAccessKey: 'secret' });

// Read in the file, convert it to base64, store to S3
var fileStream = fs.createReadStream('myarchive.tgz');
fileStream.on('error', function (err) {
  if (err) { throw err; }
});  
fileStream.on('open', function () {
  var s3 = new AWS.S3();
  s3.putObject({
    Bucket: 'mybucketname',
    Key: 'myarchive.tgz',
    Body: fileStream
  }, function (err) {
    if (err) { throw err; }
  });
});

edited Nov 27 '14 at 00:27

answered Nov 15 '14 at 20:51

CaptEmulation

4,416
2
17
14

This does not work with mp4 files. The uploaded file is not equal to the origin. – Tuyen Dinh Mar 04 '15 at 07:39
2

I'm not sure what you mean when you say "the uploaded file is not equal to the origin". I've used code similar to this to upload binary MP4 files to AWS. It almost sounds like maybe you are trying to run this code in a browser instead of NodeJS? Can you be more specific? – CaptEmulation Mar 05 '15 at 18:11
1

I'm unable to upload MP4 files using this method, as well. The file shows up in the S3 bucket, but it's corrupt and unplayable. – Gregir Oct 20 '15 at 21:15
1

Hmmmm I'm not sure why you guys are having problems. I use code like this to upload MP4 files which I can download and play again. I don't know that it makes any difference, but I am not "streaming" the files. I fully download them and play them-- but I don't think that would make any difference. I use an ubuntu host to send files to S3. You can see my code from which I created this example above here: https://github.com/CaptEmulation/soapbubble-cloud/blob/c678121e3fee472a6b3a0ca1d53c14307c68526b/soapbubble.js#L219-L251 A key difference is that this code checks if the file exists first – CaptEmulation Oct 21 '15 at 23:00
For some reason, this script buffers the whole file in memory which is a very bad thing – securecurve Dec 15 '15 at 10:29
I thought since we have the file on disk (uploaded by user to server's disk) this means the streaming will happen from disk to network, and no intermediate buffering – securecurve Dec 15 '15 at 10:30
@CaptEmulation, I tried to do what you say, but I still suffer from memory buffering, even after fully downloading the whole file. check my SO question: http://stackoverflow.com/questions/34278807/memory-buffers-fills-up-when-i-stream-upload-a-file-though-passing-a-disk-stora – securecurve Dec 15 '15 at 10:32
1

It appears you are correct and it appears to be an implementation issue within the Amazon S3 lib. S3 would need to be pausing the stream periodically in order to keep the `FileStream` from continuing to read data from disk and place in memory. You might be able to create an adapter stream implementation which tracks total bytes read but not sent and pauses itself. I'm not seeing any easy arguments to add to resolve this with glue implementation. – CaptEmulation Dec 16 '15 at 18:01

score 13 · Answer 3 · answered Dec 21 '16 at 18:17

I was able to upload my binary file this way.

var fileStream = fs.createReadStream("F:/directory/fileName.ext");
var putParams = {
    Bucket: s3bucket,
    Key: s3key,
    Body: fileStream
};
s3.putObject(putParams, function(putErr, putData){
    if(putErr){
        console.error(putErr);
    } else {
        console.log(putData);
    }
});

Upload a binary file to S3 using AWS SDK for Node.js

3 Answers3

Linked