2

So here is how my app handle image uploading and generate thumbnail

router.post('/upload', saveupload, generateThumb, saveDB)

function generateThumb (req, res, next){

    var filePath = req.files.file.path;
    var fileDir = req.files.file.dir;
    var filename = req.files.file.name;

    async.series({
        noProfile:function(callback){
            gm(filePath)
                .noProfile()
                .write(filePath, function(err){
                    callback(err);
                })
        },
        large:function(callback){
            var thumb_width = config.thumb.lg.width;
            var thumb_height = config.thumb.lg.height;
            var quality = config.thumb.lg.quality;
            var thumbName = filename.split('.')[0]+'_thumb_'+thumb_width+'x'+thumb_height+'.'+filename.split('.')[1];
            var thumbPath = path.join(fileDir, thumbName);
            gm(filePath)
                .noProfile()
                .resize(thumb_width, thumb_height)
                .quality(quality)
                .write(thumbPath, function(err){
                    callback(err)
                })
        },
        middle:function(callback){
            var thumb_width = config.thumb.md.width;
            var thumb_height = config.thumb.md.height;
            var quality = config.thumb.md.quality;
            var thumbName = filename.split('.')[0]+'_thumb_'+thumb_width+'x'+thumb_height+'.'+filename.split('.')[1];
            var thumbPath = path.join(fileDir, thumbName);
            gm(filePath)
                .noProfile()
                .resize(thumb_width, thumb_height)
                .quality(quality)
                .write(thumbPath, function(err){
                    callback(err)
                })
        },
        small:function(callback){
            var thumb_width = config.thumb.sm.width;
            var thumb_height = config.thumb.sm.height;
            var quality = config.thumb.sm.quality;
            var thumbName = filename.split('.')[0]+'_thumb_'+thumb_width+'x'+thumb_height+'.'+filename.split('.')[1];
            var thumbPath = path.join(fileDir, thumbName);
            gm(filePath)
                .noProfile()
                .resize(thumb_width, thumb_height)
                .quality(quality)
                .write(thumbPath, function(err){
                    callback(err)
                })
        }},
        function(err, obj){
            if(err){
                return next(err)
            }

            next()
        })
}

the code work well, util now I find some problem:

1:generate thumbnail this way slows the uploading speed

for every picture user uploaded, I need to generate 3 thumbnails , large, middle, small, I first save the uploaded picture to disk then pick up the picture in gm to crop metadata and generate thumbnail, this method really slow down the uploading speed, new upload can not start util the previous thumbnail is generated. what I want is keep the thumbnail generation out of the request route, so I can maintain max upload speed. any idea how?

2:Is there a better way to batch generate thumbnail

As you can see, I need 3 thumbnail, I did not see any document of gm on how to batch generate thumbnail, but I think there has to be a better way.

paynestrike
  • 4,348
  • 14
  • 46
  • 70
  • Why wait? The thumbnail generation shouldn't block anything the user can do, the only thing it can lead to is the conclusion that the uploaded image was corrupt, which you should check for before accepting it as upload anyway (trivially done client side with an ` – Mike 'Pomax' Kamermans Nov 26 '15 at 02:28
  • 1:generate thumbnail is not block anything, it just slow the upload speed which I want to improve. 2:even if I refactor the 3 generate thumbnail function to 1, and call it 3 times, nothing changed, only easier to maintain,there really should be a better way like `gm('input.jpg').batch().thumb()` – paynestrike Nov 26 '15 at 02:47
  • Your code waits for that async.series to finish before calling next(), so yeah: it's *most definitely* blocking. Instead, call next() in a non-blocking fashion first (setTimeout(0), process.tick, first call in the async schedule, whatever works for you) and *after that*, call the code that creates the thumbnails. Always form your connection response as fast as possible, and then do any additional processing outside of the connection request-response call chain. – Mike 'Pomax' Kamermans Nov 26 '15 at 03:09
  • I thought of this way too, only I a little confused, for example: picture A uploaded - savedb and return response -generate thumbnail A, then picture B uploaded -savedb and return reponse - generate thumbnail B, when B start to generate thumbnail , thumbnail A may not finish yet, I not sure what will happen then? will the process queue up or A thumbnail generation will be canceled? – paynestrike Nov 26 '15 at 03:16
  • also generate thumbnail after response is returned makes the thumbnail generation uncontrollable , I think its better to run a child process, but when use upload multiple image, even if the image is uploaded one at a time, there still a chance that I will have multiple child process, I dont know if there is a number limit of nodejs child process, and this feel more expensive. – paynestrike Nov 26 '15 at 03:28
  • there is no "child process", if you're doing this as regular JS (rather than actually using `child_process`/`spawn` to start an external process in the OS, which you shouldn't) it's a single JS thread (there is no multithreading in JS). If the user uploads their image and it's good, then the generating of thumbnails is not relevant to them getting an HTTP OK response. Send the response, then start building the thumbnails, and if they fail, have error handling in place that tracks/retries/informs/whatever is needed for *you* (not the user, they shouldn't need to care) – Mike 'Pomax' Kamermans Nov 26 '15 at 04:14
  • what happens when B start to generate thumbnail and thumbnail generation of A has not finished? – paynestrike Nov 26 '15 at 05:40
  • Nothing of note, really. Node and Express can deal with concurrent connection handling just fine. The time slice manager for Node's V8 process will simply pick whether to let operations for A's connection or B's connection run first, which is why it's so important to *first* send the response for the connection, and *then* start generating the thumbs. Thinking in terms of two different connections interfering with each other is worrying about the wrong thing when working with Node+Express. Optimize the route handler for fast responses and if needed, external further processing. – Mike 'Pomax' Kamermans Nov 26 '15 at 14:29
  • 1
    like this `res.send(); generateThumb();` ? – paynestrike Nov 27 '15 at 01:49

0 Answers0