2

I am running a real-time process from the live media stream of the webcam, which requires the FPS to stay around 30FPS. getImageData was one of my main bottlenecks, it was requiring around 22ms to extract the ImageData object from the canvas, leaving me only 11ms to do my other operations.

Here is a simple snippet of what I had before

  video = document.createElement("video");
  inputCanvas = document.createElement("canvas");
  inputCanvas.width = width;
  inputCanvas.height = height;
  inputCanvasCtx = inputCanvas.getContext("2d");
  video.width = width;
  video.height = height;
  video.srcObject = stream;

// And using the requestVideoFrameCallback API, whenever a frame is presented, this callback is triggered
  function onFrame(now, metadata){

     inputCanvasCtx.drawImage(video, 0, 0);;

     // Get frame from our canvas.
     frame = inputCanvasCtx.getImageData(0,0,inputCanvas.width, inputCanvas.height);

     // Other computations
  }

After doing this simple change to my setup, without changing anything else I managed to extract the frame in 11ms, half the time it took before, and I cannot understand why. I only converted the regular canvas to an offscreencanvas in two lines.

  video = document.createElement("video");
  inputCanvas = document.createElement("canvas");
  inputCanvas.width = width;
  inputCanvas.height = height;
  offscreen = inputCanvas.transferControlToOffscreen(); // Changed line
  inputCanvasCtx = offscreen.getContext("2d"); // Changed line
  video.width = width;
  video.height = height;
  video.srcObject = stream;

I would appreciate any insight on this matter.

Noor
  • 51
  • 1
  • 6
  • Weird, I do have the exact opposite results on my end. Using the OffscreenCanvas takes about x2 times as when using the . However I get the same (poor) result as the offscreen when I append the in the doc. And I get the same (good) perfs as the when I use the `willReadFrequently` option to create the context. Given all these I'd personally think this is because they don't manage GPU acceleration the same way for OffscreenCanvas and when there is a read-back to CPU, but I'm very surprised we have contradictory results. https://jsfiddle.net/hw1r35sy/ – Kaiido Jun 30 '22 at 02:03
  • @kaiido this is really puzzling. Did you make sure that hardware acceleration is on in your browser? And can I know your machine specifications? – Noor Jun 30 '22 at 14:42
  • 103.0.5060.53 (Official Build) (x86_64) and 105.0.5150.2 (Official Build) canary (x86_64) both on macOS, with or without "experimental web-features". And obviously with HWA on, otherwise there would be no variation at all. Do you confirm that in my test you get better results when ticking OffscreenCanvas? – Kaiido Jul 01 '22 at 02:13
  • @Kaiido I tried using "willReadFrequently " when creating the 2d context for the offscreencanvas . I noticed it reduces the performance back to 20ms from 10ms for getImageData (same value for the regular canvas). – Noor Jul 01 '22 at 13:38
  • With my fiddle, right? That's completely weird. willReadFrequently is supposed to prevent GPU acceleration, so that readbacks aren't needed since everything is always on the CPU. It makes no sense that you've got worst results (in my fiddle) when using this option, unless there is such a difference of performance between your GPU and your CPU that would make the CPU rendering actually slower than the GPU rendering + the readback. What when you change the size of the canvas? – Kaiido Jul 01 '22 at 13:48
  • And at this point I think it's fine you open an issue at https://crbug.com – Kaiido Jul 01 '22 at 13:57
  • @Kaiido I apologize. I overlooked your fiddle previously. I just tried it. When ticking willReadFrequently on and off the performance is almost negligible I can't really notice any differences. However, toggling offscreencanvas drops the value from ~30 to ~15. Appending in doc increases it by around 10. – Noor Jul 01 '22 at 14:31
  • I tested your fiddle again after a while, and the regular canvas was performing better, toggling "willReadFrequently" made the performance the same, exactly how you initially described. The performance seems to be inconsistent. In the trials where the offscreencanvas was performing much better than the regular canvas, I noticed that my GPU usage was 10% higher when I used the offscreencanvas, and toggling "willReadFrequently" on made the performance worse. I probably should open an issue. – Noor Jul 01 '22 at 15:57

0 Answers0