2

I'm using NodeJS and Multer to upload files to S3.

On the surface, everything appears to be working, the files get uploaded, and I can see them in the bucket when I log into the AWS console. However, most of the time when I follow the link to the file, the file is broken, often the file size is much smaller than the original file.

When the file reaches the server, the file size is correct if I log it, but on S3 it is much smaller. For example I just uploaded a file which is 151kb. The post request logs the file size correctly, but on S3 the file says it's 81kb.

Client side:

uploadFile = (file) ->
    formData = new FormData()
    formData.append 'file', file


    xhr = new XMLHttpRequest()
    xhr.open "POST", "/upload-image", true
    # xhr.setRequestHeader("Content-Type","multipart/form-data");

    console.log 'uploadFile'

    xhr.onerror = ->
        alert 'Error uploading file'
    xhr.onreadystatechange = ->
        if xhr.readyState is 4
            console.log xhr.responseText

    xhr.send formData

Server:

app.use(multer({ // https://github.com/expressjs/multer
      inMemory: true,
      limits : { fileSize:3000000 },
      rename: function (fieldname, filename) {
        var time = new Date().getTime();
        return filename.replace(/\W+/g, '-').toLowerCase() + '_' + time;
      },
      onFileUploadData: function (file, data, req, res) {
        var params = {
          Bucket: creds.awsBucket,
          Key: file.name,
          Body: data,
          ACL: 'public-read'
        };

        var s3 = new aws.S3();
        s3.putObject(params, function (perr, pres) {
          if (perr) {
            console.log("Error uploading data: ", perr);
          } else {
            console.log("Successfully uploaded data", pres);
          }
        });
      }
    }));

    app.post('/upload-image', function(req, res){
        if (req.files.file === undefined){
            res.end("error, no file chosen");
        } else if (req.files.file.truncated) {
            res.end("file too large");
        } else {
          console.log(req.files.file.size); //logs the correct file size
          var path = creds.awsPath + req.files.file.name;
          res.type('text/plain');
          res.write(path);
          res.end();
        };

    });

EDIT:

Setting file.buffer to the body perma onFileUploadComplete seems to work, but I have a feeling that this isn't the proper way of doing things, and may come back to bite me later. Is this approach okay, or are there issues I should be aware of doing this?

Zeeshan Hassan Memon
  • 8,105
  • 4
  • 43
  • 57
Jack Wild
  • 2,072
  • 6
  • 27
  • 39
  • `onFileUploadData` is triggered for chunks of data received. Should be `onFileUploadComplete` – hassansin Jun 17 '15 at 18:47
  • oh you are using `putObject` to upload those chunks – hassansin Jun 17 '15 at 18:48
  • aha... so perhaps if I store the file in memory, and then on complete do the put object... I'll try that, thanks for the pointer. – Jack Wild Jun 17 '15 at 20:30
  • @hassansin onFileUploadComplete I get the file, but do you know what I set as the 'Body' on the putObject params? If I put it as 'file' then that doesn't work. – Jack Wild Jun 17 '15 at 21:10
  • you can use `s3.upload` function to upload the whole file. – hassansin Jun 18 '15 at 03:24
  • @hassansin aha, onUploadComplete using file.buffer as the 'body' peram works. I guess this isn't the 'correct' way of doing things though? Couldn't work out s3.upload. – Jack Wild Jun 18 '15 at 09:02
  • In case anyone is still reading this or this other post, http://stackoverflow.com/questions/30763448/node-js-aws-s3-file-upload-how-to-get-public-url-response?rq=1, know that multer no longer supports the callbacks used here like onFileUploadData(). See the docs for the current options - https://github.com/expressjs/multer. – lsimmons Jan 14 '17 at 19:09

0 Answers0