Fast image scale down?

Question

I'm writing a "TV filter" (you know the kind, RGB bars as it zooms in), for a video file and I've been having a look at some ways of shrinking the image that retains as much detail as possible.

For testing I'm drawing the sampled image back to the screen to see the quality - in the actual filter, I'll just be sampling pixels and getting the RGB values of the resultant computed color.

I've tried three, and the Hermite filter looks good, but compared to the speed "hardware" nearest neighbour version, it's not going to be suitable for processing video.

Is there any "tricks" in JavaScript that can be used to get accelerated image shrinking like 2, but with a quality like 1 or 3?

1: Brute force: http://codepen.io/SarahC/pen/VpvWvb?editors=1010

2: Internal nearest neighbour: http://codepen.io/SarahC/pen/ryeQgN?editors=1010

3: Hermite filter: http://codepen.io/SarahC/pen/ryMNWZ?editors=1010

Here's the "hardware"? version:

function processResize(percent) {
  var size = percent * 0.01;
  var sw = canvas.width * size;
  var sh = canvas.height * size;
  ctx.drawImage(canvas2, 0, 0, sw, sh);
  ctx.drawImage(canvas, 0, 0, sw, sh, 0, 0, w, h);
}

On your brute-force, there seems to be an infinite loop + You are calling getImageData in a loop, while this operation is really slow and memory consumptive (because the browser has to de-multiply the canvas' pixels). To avoid it, call it once, then generate your tiles logic from the arrayBuffer you got. The second method should be the fastest, and at least fast enough for a video, and seems ok on my comp. Didn't took time to read third method, and I'm not sure what is the actual question. — Kaiido, Mar 06 '17 at 06:31
PS: to accelerate image processing on animated content (like a video), you may first check that you are not processing uselessly twice the same frame (use the `currentTime` property of your video element), and use the hardware shrinking method, to process on a smaller version of the image data. — Kaiido, Mar 06 '17 at 06:32

score 1 · Accepted Answer · 2017-03-08T03:01:56.540

I am not entirely sure from the description what you try to achieve, but from the codepens it seems as you try to create a mosaic effect.

You can use the built-in interpolation setting of the canvas context to use nearest-neighbor by turning image smoothing off, then draw the image to a small size representing how many "blocks" you want. Then draw back that version to full size again:

// blocks = initial number of pixels (video aspect is usually 16:9 so you may want
// to calculate a separate values for height.
var blocks = 24;

// draw initial size representing "blocks"
ctx.drawImage(video, 0, 0, blocks, blocks);

// turn off image smoothing (see below for prefixing)
// This uses nearest neighbor
ctx.imageSmoothingEnabled = false;

// enlarge the mosaic back to full size
ctx.drawImage(c, 0, 0, blocks, blocks, 0, 0, c.width, c.height);

Video Example

(the video may take a few seconds to load...)

var ctx = null;
var blocks = 24;
var video = document.createElement("video");
video.preload = "auto"; video.muted = video.autoplay = video.loop = true;
video.oncanplay = function() {  // initialize for demo
  if (!ctx) {
    c.width = this.videoWidth;
    c.height = this.videoHeight;
    ctx = c.getContext("2d");

    document.querySelector("input").oninput = function() {blocks = +this.value};
    requestAnimationFrame(loop);
  }
}
video.src = "//media.w3.org/2010/05/sintel/trailer.mp4";

function smoothing(state) {
  ctx.oImageSmoothingEnabled = ctx.msImageSmoothingEnabled =
  ctx.mozImageSmoothingEnabled = ctx.webkitImageSmoothingEnabled =
  ctx.imageSmoothingEnabled = state;
}

function loop() {
  smoothing(true);  // improve quality of first step
  ctx.drawImage(video, 0, 0, blocks, blocks);

  smoothing(false); // mosaic step
  ctx.drawImage(c, 0, 0, blocks, blocks, 0, 0, c.width, c.height);
  
  // loop and throttle to 30 fps
  requestAnimationFrame(function() {requestAnimationFrame(loop)});
}

<label>Blocks: <input type=range min=8 max=128 value=24></label><br>
<canvas id=c></canvas>

Thank you for the example! My problem was the downsampling of the image threw away pixels rather than averaged them - so details like lines would mostly vanish and what remained would get aliased (example 2 in my code shows a white diagonal line get aliased away when you increase the block size) . The DIY downscaling worked much better - the white line remained visible as the block size increased, but was much too slow. From what you've demo'd here, a better downsampling isn't possible realtime. — SarahC, Mar 07 '17 at 04:24
@SarahC ah, in that case you can improve the quality slightly by turning smoothing on for the first down-scale, and off for the final mosaic (I'll update answer). — , Mar 08 '17 at 02:59
@SarahC in the future there will likely be a quality *type* option for the smoothing (currently defined as `imageSmoothingQuality` in the spec) but is currently only available in Chrome/Opera. This allow you to set the quality to "high" (Lanczos or something similar) and will preserve more details. At the moment we only have bi-linear so if you need more quality than this you'd need to downscale in steps (see [here](http://stackoverflow.com/a/17862644/1693593) for an example). The browser is usually fast enough for real-time processing these extra steps. — , Mar 08 '17 at 03:07

Fast image scale down?

1 Answers1

Video Example