Transform upload with NodeJS Multer

Question

I'm currently implementing a file/image upload service for my users. I want to transform these images (resize/optimize) before uploading to my s3 bucket.

What I'm currently doing: Using a multipart form on my frontend (I think the actual implementation doesn't matter here..) and the multer and multer-s3 packages on my backend.

Here my implementation stripped down to the important parts.

// SETUP
var multer = require('multer');
var s3 = require('multer-s3');
var storage = s3({
    dirname: 'user/uploads',
    bucket: auth.aws.s3.bucket,
    secretAccessKey: auth.aws.s3.secretAccessKey,
    accessKeyId: auth.aws.s3.accessKeyId,
    region: auth.aws.s3.region,
    filename: function (req, file, cb) {
        cb(null, Date.now());
    }
});
var upload = multer({storage: storage}).single('img');

// ROUTE
module.exports = Router()
    .post('/', function (req, res, next) {
        upload(req, res, function (err) {
            if (err) {
                return res.status(401).json({err: '...'});
            }
            return res.json({err:null,url: '..'});
        });
    });

What I want to do: transform the image before uploading it. I'm not sure if I need to use multer/busboy here or I can just do it with NodeJS (thus I've tagged NodeJS and express as well).

So my question is: where can I intercept the upload and transform it before uploading it to my S3 bucket?

Well, you could save the file in some temp folder before uploading to S3? If you don't want to save the files you could do this action in memory, that depends how large images are, this approach would not suit for larger files. — Risto Novik, Feb 21 '16 at 19:51
@RistoNovik well I probably could do this. But that sounds very expensive/inconvenient. — boop, Feb 21 '16 at 19:52
You can transform the image on the frontend before sending it to the backend. — Solomon Ayoola, Feb 21 '16 at 19:54
@AyoolaSolomon true this would be the first step for resizing anyway, but you shouldn't rely only on a frontend. — Risto Novik, Feb 21 '16 at 19:56
@AyoolaSolomon Yeah no. I could resize it; but I can't imagine I could optimize it perfectly on the frontend. Even if I could: It's not guaranteed to be done; someone just could `POST` it without these transforms and I want to avoid that in any case. — boop, Feb 21 '16 at 19:56
Just another approach would be AWS Lambdas, which triggers the specific chain of image transforms after the file has been written to S3. Also, this is a quite scalable solution. The image transforms usually does not fit into Node.js single threaded model. — Risto Novik, Feb 21 '16 at 19:59
Besides all the other listed approaches there are cloud image services which do all the transforms in real time like Cloudinary. — Risto Novik, Feb 21 '16 at 20:01
@Brettetete I totally agree with that. I was also thinking you can transform the image before setting the s3 params. So you can save the optimized version of the image to S3. — Solomon Ayoola, Feb 21 '16 at 20:03
Thanks for all the listed approaches. I were aware of these services but they are connected with additional costs which I **have to** avoid. — boop, Feb 21 '16 at 20:04
@Brettetete I think you can also follow the path of using Cloudinary as suggested above as it provides some image transformation and processing out of the box for you. Take a look at this http://cloudinary.com/documentation/node_image_manipulation — Solomon Ayoola, Feb 21 '16 at 20:06
@AyoolaSolomon **As I said**: I can't use these services since they are connected with additional costs. — boop, Feb 21 '16 at 20:07
Well then you all have to do all the transforms in memory, read and write in buffer and hold the reference to the instance, it's quite easy to get some mem leaks and under load you have to implement some sort of limiter. Depends quite what instance/machine are you using. — Risto Novik, Feb 21 '16 at 20:10
@Brettetete I think that the most reliable option is to use the Task Queue: 1s) to save the file locally; 2) create a task to optimize the image; 3) at the end of optimization to create a task to download files to c3; 4) once the file download is complete remove the local file and send a message to the user — stdob--, Feb 21 '16 at 21:14

Gregor Menih · Answer 1 · 2017-07-10T11:15:06.727

9

Not sure if you're still looking for an answer to this, but I had the same problem. I decided to extend the multer-s3 package.

I've opened a pull request to the original repository, but for now, you can use my fork.

Here's an example of how to use the extended version:

  var upload = multer({
    storage: multerS3({
      s3: s3,
      bucket: 'some-bucket',
      shouldTransform: function (req, file, cb) {
        cb(null, /^image/i.test(file.mimetype))
      },
      transforms: [{
        id: 'original',
        key: function (req, file, cb) {
          cb(null, 'image-original.jpg')
        },
        transform: function (req, file, cb) {
          cb(null, sharp().jpg())
        }
      }, {
        id: 'thumbnail',
        key: function (req, file, cb) {
          cb(null, 'image-thumbnail.jpg')
        },
        transform: function (req, file, cb) {
          cb(null, sharp().resize(100, 100).jpg())
        }
      }]
    })
  })

EDIT: My fork is also now available via npm under the name multer-s3-transform.

edited Jul 10 '17 at 11:15

answered Oct 25 '16 at 10:16

Gregor Menih

5,036
14
44
66

I am using this code but this error popping up. ```Cannot find name'sharp'.```. do I have to install ```sharp``` npm package or this is builtin? – Usama Tahir Dec 21 '18 at 14:25
You have to implement the transformation... In this example, I use sharp, but you can use anything you want. – Gregor Menih Dec 21 '18 at 14:26
1

can you provide full working example? I have installed ```sharp``` package but it is giving me error that ```sharp(...).jpg is not a function``` – Usama Tahir Dec 21 '18 at 14:31
I also had sharp(...).jpg is not a function issues. The way I solved them is that I assume sharp does not have a jpg function any longer. I used the .png() function instead, which I actually prefer to jpgs anyways. If you still wish to retain the jpg format, I would recommend seeing what the latest on sharp entails via: https://www.npmjs.com/package/sharp – Daniel Brown Dec 26 '18 at 17:40
For me its not working, no error occur, but file is not uploading and not goes inside this function. `var upload = multer({........}).single("file"); upload(req, res, function (err, filePath) { console.log(err, "Error"); console.log(filePath, "*************"); });` – jones May 13 '19 at 06:18
omg you have made a whole npm package to handle this. Thank you SO much! This should not just be accepted but get a million upvotes! – Mattia Rasulo Nov 27 '19 at 22:58

andrewhl · Answer 2 · 2017-09-04T01:13:57.580

I've tried using @ItsGreg's fork, but couldn't get it to work. I managed to get this behaviour working by using multer-s3 standard configuration, and inside my file upload endpoint, i.e.,

app.post('/files/upload', upload.single('file'), (req, res) => {...})

I am retrieving the file using request, and passing the Buffer to sharp. The following works (and assumes you are using ~/.aws/credentials):

    let request = require('request').defaults({ encoding: null });
    let dataURI = `https://s3.amazonaws.com/${process.env.AWS_S3_BUCKET}/${image.defaultUrl}`;
    request.get(dataURI, function (error, response, body) {
        if (! error && response.statusCode === 200) {
            let buffer = new Buffer(body);
            const sizes = ['thumbnail', 'medium', 'large'];
            sizes.forEach(size => {
                sharp(buffer)
                    .resize(image.sizes[size])
                    .toBuffer()
                    .then(data => {

                        // Upload the resized image Buffer to AWS S3.
                        let params = {
                            Body: data,
                            Bucket: process.env.AWS_S3_BUCKET,
                            Key: `${image.filePath}${image.names[size]}`,
                            ServerSideEncryption: "AES256",
                        };

                        s3.putObject(params, (err, data) => {
                            if (err) console.log(err, err.stack); // an error occurred
                            else console.log(data);           // successful response
                        });
                    })
            })
        }
    });

Transform upload with NodeJS Multer

2 Answers2

Linked