3

I've noticed a performance problem that many people are writing a <video> image to the <canvas> and then reading the pixels from the <canvas> and then rewriting the data back to the <canvas> to do real time image processing in HTML5/JavaScript.

Obviously the correct way would be to read the <video> image data and then process the pixels in javascript and only write to the <canvas> once.

It appears that the getUserMedia <video> API doesn't support direct access to the image data even on an event, perhaps for some performance reason??!? Does this require a write to the canvas if you want to read the actual pixel information? I don't mind if the data is compresses it would be nice to have it in some form. I did notice that there is a reference to a video.src however it appears that getUserMedia doesn't include a video.data thats usable.

https://developer.mozilla.org/en-US/docs/Web/API/Navigator/getUserMedia https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D

Here is some example code that works however I would like to speed it up by not writing to the <canvas> and read directly from the <video> data:

function DisplayCanvas (video) {
  var delay = 1500; // ms
  timer = setInterval(
    function () {
      // display is a canvas context
      display.drawImage(video, 0, 0, 640, 480);
      data = display.getImageData(0,0,640,480);
      idata = ImageProcessing(data);
      display.putImageData(idata,0,0);
    }, delay);
  }

Is there anyway to simplify drawImage() first problem to just one line with.

// the problem is that there isn't a video.data 
display.putImageData(ImageProcessing(video.data),0,0);
Asher
  • 2,638
  • 6
  • 31
  • 41
  • There's some discussion at http://stackoverflow.com/questions/12130475/get-raw-pixel-data-from-html5-video?rq=1. – deltab Aug 28 '15 at 17:54
  • Thanks, delta, interesting example however from what I can tell in that example from Mozilla https://developer.mozilla.org/samples/video/chroma-key/index.xhtml they are getting the data from the canvas not the video stream. So they copy a frame from the canvas with frame = this.ctx1.getImageData(0, 0, this.width, this.height); process it and then rewrite it back to the canvas rather then getting the data from the video stream itself. Also we are looking for something that works on both firefox and chrome (for some reason it didn't work for me in chrome but did in firefox) – Asher Aug 28 '15 at 18:02

2 Answers2

1

I don't think you can spare this step, you need to transform the stream into something, and that's what the drawImage does.

I think the problem resides in the pixel manipulation stage. You have to do it on the gpu if you want speed. To do that you need to render the video to a webGL texture and manipulate it via shaders.

With that your bottleneck will be (I think) the speed at which the browser pushes the video to the gpu.

estani
  • 24,254
  • 2
  • 93
  • 76
0

Canvas's .getImageData & .putImageData are the only way of manipulating pixels on live video tags on the client side.

They not GPU accelerated so they are quite slow.

On a reasonable desktop, you should be able to sample + modify + rewrite the image, although you might have dropped frames.

On lesser machines the process will likely fail.

markE
  • 102,905
  • 11
  • 164
  • 176
  • We are seeing about 20 frames a second at 640x480 processing pixel by pixel everything in Javascript which is quite fast when you think about it. However, this requires that we write to the `` essentially twice, first with the raw data then we read data then we process the data then we rewrite the data. I think it would be faster to just process the data and write just once if possible and our frame rate maybe would increase by 25% or more? – Asher Aug 28 '15 at 18:10
  • @asolr, Pixel manipulation requires both writes: original video frame & modified image. There really is no workaround other than pre-processing. ;-) – markE Aug 28 '15 at 18:24