How to reduce getImageData's memory usage when using canvas to edit video?

Question

I've been trying to use a canvas to edit video by drawing to a offscreen canvas and then using getImageData to do some work and then putting that onto my onscreen canvas. It works, but even with a small 480x360 video, Chrome memory usage keeps increasing until it crashes (takes about ten minutes on my machine, less with larger video). The issue seems to be better in Firefox, but still suffers from large memory usage.

I realize that each getImageData call takes ~3MB of memory, but even then I feel like there should be a way to get Chrome to use less than 1GB of memory. I've lowered the framerate, which helps but does not solve the issue. Is there anything I can do to ensure that the imageData memory is freed in a more timely fashion?

<!DOCTYPE html>
<meta charset='utf-8'>
<html>
    <head>
        <title>Video Test</title>
    </head>
    <body>
        <canvas id='display'>
        </canvas>
        <script type='text/javascript'>
            var canvas = document.getElementById('display');
            var context = canvas.getContext('2d');

            var bCanvas = document.createElement('canvas');
            var bContext = bCanvas.getContext('2d');

            var video = document.createElement('video');
            video.src = 'VIDEO HERE';
            video.autoplay = true;
            video.loop = true;
            var last;
            var interval = 35;
            video.addEventListener('play', function() {
                canvas.width = video.videoWidth;
                canvas.height = video.videoHeight;
                bCanvas.width = video.videoWidth;
                bCanvas.height = video.videoHeight;
                last = performance.now();
                window.requestAnimationFrame(draw);
            }, false);

            function draw(time) {
                if(time - last > interval) {
                    bContext.drawImage(video,0,0,bCanvas.width,bCanvas.height);
                    var imageData = bContext.getImageData(0,0,bCanvas.width,bCanvas.height);
                    context.putImageData(imageData,0,0);
                    last = time;
                }
                window.requestAnimationFrame(draw);
            }

        </script>
    </body>
</html>

This might be worth a look: http://stackoverflow.com/questions/742623/deleting-objects-in-javascript — enhzflep, Nov 13 '15 at 07:54
@enhzflep I've tried using delete on imageData.data just to try it, but nothing is accomplished. There are not any references to imageData once draw returns, so it shouldn't matter to the GC. The memory is eventually freed, it's just that the GC in Chrome is not being very timely... — Josh, Nov 13 '15 at 08:01
Rather than calling it on `imageData.data` member, I would investigate the effects of using `delete` on the `imageData` object. Failing that, I've no other suggestions unfortunately. — enhzflep, Nov 13 '15 at 08:05
1) Just handle play and pause events to update a `isPlaying` flag, and update the canvas size when required (you could use also video.playing property). 2) have the draw loop run all the time and update only if `isPlaying`. I fear you might trigger many simultaneous draw loops with this code, which might pile-up (especially when looping) and then use too much memory. — GameAlchemist, Nov 13 '15 at 09:49
What are you doing with the imageData? Are you sure you need it? `drawImage` and `globalCompositeOperation` are way lighter, maybe you could simply use these method/property instead. — Kaiido, Nov 13 '15 at 10:13
Ditto @Kaiido's comment. What are you doing with the pixels & use compositing if possible. Otherwise the only way to reduce `.getImageData` memory usage is to call it less often. — markE, Nov 13 '15 at 16:13
@Kaiido I am trying to achieve video transparency much in the same way as the solution here http://stackoverflow.com/questions/5055962/how-to-create-an-h264-video-with-an-alpha-channel-for-use-with-html5-canvas — Josh, Nov 13 '15 at 16:38
@Josh, The code in linked question is quite dirty, it seems it assumes that the canvas won't have the same size as the video to be drawn, and uses the missing part as the alphaChannel... I'm not sure I quite get why they did something like that... If you only want transparency, then you can just set the `context.globalAlpha` property and draw your frames with `drawImage()` method. — Kaiido, Nov 14 '15 at 06:42
@Kaiido The effect that I want to achieve is individual pixel transparency. Using context.globalAlpha will set the transparency for the entire video, what I want is certain sections to be transparent, in an effect like [chroma key](https://en.wikipedia.org/wiki/Chroma_key). The example question is using the bottom part of the video as a mask for the top half of the video. Unfortunately h264 does not have alpha support afaik, thus the need for this workaround. — Josh, Nov 14 '15 at 18:38
Is the video short and looping? If so, it might be beneficial to keep all the frame in memory rather than recreate them. — Alexander O'Mara, Nov 15 '15 at 07:22
@Josh, then I'm afraid gCO won't help either, or at least, not directly. Because, one solution I think of, would be to first draw your frames on a smaller canvas, do the calculations/transparency on this one, and then redraw the now-transparent at a normal scale, then setting gCo to `"source-in" before redrawing the original frame, you should keep only the wanted parts. Didn't tested it yet, and I may post it as an answer tomorrow if I've got time. Also, maybe of some interest, there is this MDN blog post https://hacks.mozilla.org/2011/12/faster-canvas-pixel-manipulation-with-typed-arrays/ — Kaiido, Nov 15 '15 at 09:51
@Kaiido The problem is that I've compromised quality a decent amount already (the video is displayed at 960x720) and I shrunk the source to 480x360, basically thinking the same thing you are. I don't think I can compress it further without drastic quality loss. — Josh, Nov 15 '15 at 19:24
@Josh, I'm not talking about compromising the video quality, but the chroma detection. I've added an answer since I'm pretty sure you weren't thinking the same thing as I was. — Kaiido, Nov 16 '15 at 02:43

Kaiido · Answer 1 · 2018-10-09T07:48:59.913

Since what you are trying to achieve is a chroma key effect, you could do the chroma detection on a down-sampled frame, set the transparency on it then redraw it at a normal scale on the output canvas.

Then, thanks to the globalCompositeOperation property of your output context set to "destination-in", you can draw only the non-transparent part of your original frame, keeping its original quality :

// Define our canvases
var output = document.createElement('canvas'),
  ctx = output.getContext('2d'),
  buffer = output.cloneNode(),
  buf = buffer.getContext('2d');

document.body.appendChild(output);

var threshold = colorThreshold.value,
  color = hexToRgb(colorInp.value.split('#')[1]);

var lastCall = 0;

function draw() {
  requestAnimationFrame(draw);
  // if the video is still at the same frame, we don't need to process anything
  if (video.currentTime === lastCall)
    return;
//  video.pause();
  lastCall = video.currentTime;

  // clear our output canvas
  ctx.clearRect(0, 0, output.width, output.height);
  ctx.drawImage(video, 0, 0, output.width, output.height);
  // draw a downsampled frame on the buffer canvas
  buf.drawImage(video, 0, 0, buffer.width, buffer.height);
  // get this downsampled canvas's imageData
  var image = buf.getImageData(0, 0, buffer.width, buffer.height),
    data = image.data;

  var t = threshold / 2;

  // loop through the imageData pixels
  for (var i = 0; i < data.length; i += 4) {
    // for a correct Chroma key, this should be improved
    if ((color[0] - t) <= data[i] && data[i] <= (color[0] + t) &&
      (color[1] - t) <= data[i + 1] && data[i + 1] <= (color[1] + t) &&
      (color[2] - t) <= data[i + 2] && data[i + 2] <= (color[2] + t)) {
      // set the alpha channel to 0
      data[i + 3] = 0;
    }
  }
  // redraw our now-tranparent image on the buffer
  buf.putImageData(image, 0, 0);
  // set our context's gCO to destination-in ...
  ctx.globalCompositeOperation = 'destination-in';
  // resample the buffer to a normal scale (bad quality)
  ctx.drawImage(buffer, 0, 0, output.width, output.height);
  // reset the context's gCO
  ctx.globalCompositeOperation = 'source-over';

}

colorThreshold.addEventListener('input', function() {
  threshold = this.value;
});
colorInp.addEventListener('input', function() {
  color = hexToRgb(this.value.split('#')[1]);
});
cutQ.addEventListener('input', function() {
  buffer.width = (output.width / 100) * this.value;
  buffer.height = (output.height / 100) * this.value;
});
video.addEventListener('loadedmetadata', function() {
  output.width = this.videoWidth;
  output.height = this.videoHeight;
  buffer.width = (output.width / 100) * cutQ.value;
  buffer.height = (output.height / 100) * cutQ.value;
  draw();
});

// convert our input's value to rgb
function hexToRgb(hex) {
  var bigint = parseInt(hex, 16),
    r = (bigint >> 16) & 255,
    g = (bigint >> 8) & 255,
    b = bigint & 255;
  return [r, g, b];
}

canvas {
  width:100%;
  border: 1px solid;
  background: url('data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAGUlEQVQYlWP4z8DwHx0zYANDQeEgcw5FCgHKJ2OdLLDjkAAAAABJRU5ErkJggg=='), repeat;
}

Color to pick :
<input type="color" value="#30f062" id="colorInp" /><br>
Cut quality :
<input type="range" min="10" max="100" step=".5" value="80" id="cutQ" /><br>
Color Threshold :
<input type="range" min="0" max="255" step=".5" value="166" id="colorThreshold" /><br>
<video id="video" style="position:absolute;visibility:hidden;z-index:-1" autoplay="" muted="true" loop="true" src="https://dl.dropboxusercontent.com/s/1jp0f76yvzuucj7/L0ckergn0me-PixieGreenScreen446_512kb.mp4" crossorigin="anonymous"><!--CC BY-NC-SA 2.0 http://chris.pirillo.com/--></video>

How to reduce getImageData's memory usage when using canvas to edit video?

1 Answers1

Linked