5

Basically I want to be able to perform effectively this same code:


const video = document.getElementById('video');
const canvas = document.getElementById('canvas');
const context = canvas.getContext('2d');

const draw = () => {
  context.drawImage(video, 0, 0);
  requestAnimationFrame(draw);
}

video.onplay = () => {
  requestAnimationFrame(draw);
}

only using an offscreen canvas. I can send images over messages to the worker the offscreen canvas is on, but not video as it's directly tied to an HTMLElement. Is there currently a way to somehow still render video data or a MediaStream in an offscreen canvas?

Jacob Greenway
  • 461
  • 2
  • 8
  • 15

1 Answers1

21

You can send frames of a video to an OffscreenCanvas in a Web Worker by modifying your script with the following changes:

const worker = new Worker('my-worker.js');
const video = document.getElementById('video');
const stream = video.captureStream();
const [track] = stream.getVideoTracks();
const imageCapture = new ImageCapture(track);
const canvas = document.getElementById('canvas');
const offscreen = canvas.transferControlToOffscreen();

worker.postMessage({ offscreen }, [offscreen]);

const draw = () => {
  imageCapture.grabFrame().then(imageBitmap => {
    worker.postMessage({ imageBitmap }, [imageBitmap]);
  });

  requestAnimationFrame(draw);
};

video.onplay = () => {
  requestAnimationFrame(draw);
};

my-worker.js

let canvas;
let context;

addEventListener('message', event => {
  if (event.data.offscreen) {
    canvas = event.data.offscreen;
    context = canvas.getContext('2d');
  } else if (event.data.imageBitmap && context) {
    context.drawImage(event.data.imageBitmap, 0, 0);
    // do something with frame
  }
});

References

Patrick Roberts
  • 49,224
  • 10
  • 102
  • 153
  • Thanks for the reply, but I guess I was looking for a way to offload the requestAnimationFrame calls by using the offscreen canvas since requestAnimationFrame is throttled by browsers when the tab is blurred. This solves my original question, but is there any way to get the video to the offscreen canvas without a timer/requestAnimationFrame calls? – Jacob Greenway Jun 25 '19 at 06:33
  • @JacobGreenway depends on the `src` of the video. But `requestAnimationFrame()` is also throttled in web workers when the tab that spawned them loses focus, so you'll need to get a little more clever than just running your original code in a web worker. – Patrick Roberts Jun 25 '19 at 06:43
  • Unfortunately still only supported in Blink browsers, for FF you could use the [createImageBitmap](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/createImageBitmap) method passing in a – Kaiido Jun 25 '19 at 07:42
  • what would be the benefit of doing something like this, if the worker isn't handling the animation? – hoodsy Mar 04 '20 at 23:15
  • 1
    @hoodsy `grabFrame()` and `postMessage()` called with a [Transferable](https://developer.mozilla.org/en-US/docs/Web/API/Transferable) object are not very expensive computations in the DOM thread, so it has a lot of free time to handle user events and repaints, while the expensive work of `drawImage()` and other canvas context methods are pushed to a worker thread, where it's not negatively impacting the responsiveness of the web page. – Patrick Roberts Mar 15 '21 at 16:44
  • @PatrickRoberts transferControlToOffscreen has a low browsers support – Alexufo Jun 25 '21 at 02:07