2

Backgroud

I'm now processing on the client select image.

I want to do two actions on that image, and outputs the base64-encoded string.

  1. If the image size has a width or height larger than 1000, resize it.
  2. Compress the image with jpeg of quality 0.5.

So now I will do the below in the main script:

$(function() {

  $('#upload').change(function() {

    var URL = window.URL || window.webkitURL;

    var imgURL = URL.createObjectURL(this.files[0]);

    var img = new Image();

    img.onload = function() {

      var canvas = document.createElement('canvas');
      var ctx = canvas.getContext('2d');
      var w0 = img.width;
      var h0 = img.height;
      var w1 = Math.min(w0, 1000);
      var h1 = h0 / w0 * w1;

      canvas.width = w1;
      canvas.height = h1;

      ctx.drawImage(img, 0, 0, w0, h0,
        0, 0, canvas.width, canvas.height);

      // Here, the result is ready, 
      var data_src = canvas.toDataURL('image/jpeg', 0.5);
      $('#img').attr('src', data_src);

      URL.revokeObjectURL(imgURL);
    };

    img.src = imgURL;

  });

});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<input id="upload" type="file" accept="image/*" />
<img id="img" />

The Problem

But still, my code will work on a mobile, where the above procedure(resize and compress) can not work out soon. It causes the GUI stop response for a moment.

I want the procedure work in another thread, using web worker. So it won't block the UI, so the user experience would be better.

Now comes the problem, it seemed the web worker cannot deal with a canvas, how can I work around this?

Alfred Huang
  • 17,654
  • 32
  • 118
  • 189
  • How big are your images so loading blocks the UI ? I think that the only way to use a webworker for this kind of task would be to extract the imageData of your canvas and send it to the webworker where you will have to rewrite yourself a scaling/compression algorithm... The getImageData/putImageData is at least as CPU consumptive as what you are doing. Are you sure the problem comes from this function ? – Kaiido Nov 28 '15 at 13:22
  • 1
    Using webworkers will take the delay off the UI thread. This will decrease the UI delays as long as the device has good support for webworkers. But using webworkers will increase the overall burden because the image data must be both marshalled to the webworkers and then marshalled back when their job is complete. – markE Nov 28 '15 at 17:47

1 Answers1

6

Some Event driven coding

Saddly Web workers are not yet ready with browser support.

Limited support for toDataURL in web workers means another solution is needed. See MDN web worker APIs (ImageData) mid way down the page, only for Firefox at the moment.

Looking at your onload you have all the heavy duty work done in one blocking call to onload. You are blocking the UI during the process of creating the new canvas, getting its context, scaling, and toDataURL (don't know what revokeObjectURL does). You need to let the UI get a few calls in while this is happening. So a little event driven processing will help reduce the glitch if not make it unnoticeable.

Try rewriting the onload function as follows.

// have added some debugging code that would be useful to know if
// this does not solve the problem. Uncomment it and use it to see where
// the big delay is.
img.onload = function () {
    var canvas, ctx, w, h, dataSrc, delay; // hosit vars just for readability as the following functions will close over them
                                    // Just for the uninitiated in closure.
    // var now, CPUProfile = [];  // debug code 
    delay = 10; // 0 could work just as well and save you 20-30ms
    function revokeObject() { // not sure what this does but playing it safe
        // as an event.
        // now = performance.now(); // debug code
        URL.revokeObjectURL(imgURL);
        //CPUProfile.push(performance.now()-now); // debug code
        // setTimeout( function () { CPUProfile.forEach ( time => console.log(time)), 0);
    }
    function decodeImage() {
        // now = performance.now(); // debug code
        $('#img').attr('src', dataSrc);
        setTimeout(revokeObject, delay); // gives the UI a second chance to get something done.
        //CPUProfile.push(performance.now()-now); // debug code
    }
    function encodeImage() {
        // now = performance.now(); // debug code
        dataSrc = canvas.toDataURL('image/jpeg', 0.5);
        setTimeout(decodeImage, delay); // gives the UI a second chance to get something done.
        //CPUProfile.push(performance.now()-now); // debug code
    }
    function scaleImage() {
        // now = performance.now(); // debug code
        ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
        setTimeout(encodeImage, delay); // gives the UI a second chance to get something done.
        //CPUProfile.push(performance.now()-now); // debug code
    }
    // now = performance.now(); // debug code
    canvas = document.createElement('canvas');
    ctx = canvas.getContext('2d');
    w = Math.min(img.width, 1000);
    h = img.height / img.width * w;
    canvas.width = w;
    canvas.height = h;
    setTimeout(scaleImage, delay); // gives the UI a chance to get something done.
    //CPUProfile.push(performance.now()-now); // debug code
};

setTimeout allows the current call to exit, freeing up the call stack and allowing the UI to get its mitts on the DOM. I have given 10ms, personally I would start with 0ms as call stack access is not blocked, but I am playing it safe

With luck the problem will be greatly reduced. If it still remains an unacceptable delay then I can have a look at the CPU profile and see if a solution can not be found by targeting the bottle neck. My guess is the toDataURL is where the load is. If it is, a possible solution is to find a JPG encoder written in javascript that can be converted to an event driven encoder.

The problem is not how long it takes to process the data, but how long you block the UI.

Blindman67
  • 51,134
  • 11
  • 73
  • 136
  • It's very detailed and pointing out a reasonable solution, thank you. – Alfred Huang Nov 29 '15 at 01:45
  • But is there any way to broke the compression? It may take several seconds. – Alfred Huang Nov 29 '15 at 01:46
  • 1
    @Blindman67, you can think of `URL.revokeObjectURL()` as a `free cache` or `delete` for the blob that was generated by the `createObjectURL` method. By the way, @fish_ball you could also try to do this computation using a `FileReader.readAsDataURL`, which will load the file asynchronously instead of the synchronous `createObjectURL()`. – Kaiido Nov 29 '15 at 03:24
  • **Fish_ball** Compression? Image to JPG or Resize?. The resize will be done by the GPU (if available). It will only be noticeable if the image is larger than the RAM available to the GPU. For the JPG compression that is done by a native function it can't be done faster unless you use a lower level language. @Kaiido thanks. Revoke then is trivial until GC kicks in and that's rather well done on most browsers. – Blindman67 Nov 29 '15 at 04:03
  • 1
    @Blindman67 `revokeObjectURL` is not really trivial : the blob will be cached in browser even if you close the tab that created the ObjectURL, in fact I think it's only cleared when you quit the browser, I don't think GC will collect it. So `revokeObjectURL` should not be forgotten. But I do agree it has nothing to do with OP's issue. – Kaiido Nov 29 '15 at 05:07
  • @Kaiido, what about sending the file blob the a web worker then use a js written compression? Is there any library to compress jpeg and png as a binary array? – Alfred Huang Nov 29 '15 at 13:53
  • @fish_ball, Once again, and as also said by @markE, the getImageData+ putImageData operations needed by the webworkers are more expansive than a toDataURL(). Did you tried the FileReader? Are you sure the bottleneck is in the compression? I was able to do a `toDataURL('image/jpeg')` of a few Mb image in less than few ms on every mobile phones I have. Your "several seconds" seems really strange to me. – Kaiido Nov 29 '15 at 14:24
  • @Kaiido for example, see this question http://stackoverflow.com/a/32774035/2544762, the compress operation make take a long time. – Alfred Huang Nov 29 '15 at 14:45
  • You don't need to compress to an image format. Any compression format will do. Compress the raw pixel data, send to the server, then uncompress and create the image on the server where you have many more options. Search gitHub they have lots of compression utilities. – Blindman67 Nov 29 '15 at 14:55