20

I would like to get the raw data (TypedArray or something) from video element and manipulate them with JavaScript.

Currently I create a new canvas, draw the video into canvas and then get the image data.

ctx.drawImage(myVideo);
var data = ctx.getImageData(0, 0, w, h).data;

It works fine, but it drains a CPU (putting the video to canvas and copying back from canvas) and it creates a lot of garbage (about 50 MB each second). Is there any other simpler solution? It would be great if I could pass my own buffer to getImageData(...).

BTW. drawing video with WebGL and loading it back from GPU is not any faster :( http://jsperf.com/getting-raw-data-from-video

Ivan Kuckir
  • 2,327
  • 3
  • 27
  • 46
  • This is the easy way to do it. You might be able to get more performance via WebGL. You can see how https://github.com/brianchirls/Seriously.js does it for video filters. – forresto Sep 20 '12 at 11:09
  • Hey, thanks for link. That library edits the raw data in Fragment Shader. But I need to edit it from JavaScript :( – Ivan Kuckir Sep 20 '12 at 14:03

3 Answers3

3

Please read

https://www.scirra.com/blog/76/how-to-write-low-garbage-real-time-javascript

Because you don't actually show any code, mention the browser or give out information about the video (resolution, FPS, encoding) it is impossible to tell why your code is slow or why it creating show much garbage. Real-time video effects with Javasrcipt are possible with some resolution constrain.

Here is an Mozilla example of real-time video filtering using .

https://developer.mozilla.org/samples/video/chroma-key/index.xhtml https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas

You won't get any raw access to video data without blitting a video frame on <canvas> first. However, I believe this operation should be HW accelerated as it happens all in GPU memory. Downloading pixels down from GPU manipulating them in Javascript and then uploading back to the display memory might be the slowest step for high resolution.

Optimization tips

root
  • 1,812
  • 1
  • 12
  • 26
Mikko Ohtamaa
  • 82,057
  • 50
  • 264
  • 435
  • My question is not about bowser, it is about W3C specification. I showed you the code. The problem is, that HTML5 canvas context function getImageData produces a new ArrayBufferView, which has 4 bytes for each pixel. That's why I am looking for some other solution. – Ivan Kuckir Sep 20 '12 at 23:21
  • 1
    Hey I am getting raw access to video data - I even wrote the code how I do that... – Ivan Kuckir Sep 20 '12 at 23:26
  • 1
    BTW. Mozilla example reads raw data the same way as I do (getImageData(...)). No improvements there. – Ivan Kuckir Sep 20 '12 at 23:30
  • Chrome implements incremental garbage collector which should help with fast object creation; also you can tune garbage collector settings on the other browsers. You do not still mention which browser you use to consume 50 MB / sec and with which kind of frame. Y – Mikko Ohtamaa Sep 21 '12 at 07:30
  • I updated the question. With raw video data access I mean accessing video.getImageData() directly, without blitting it to the canvas first. As you can see getImageData() is only way to manipulate the data currently. But you are right the API does not give you ability to recycle your own buffer, but it should not create a new buffer either. Are you sure the garbage comes from the pixel data and not from your other code? Maybe you keep copies of the pixel data pointer around in your Javascript (scoping done wrong) and pixel data references are never reclaimed. – Mikko Ohtamaa Sep 21 '12 at 07:39
  • Hey, all my garbage is collected. Used memory starts at 30 MB, it gets to 300 MB after 4 seconds, then it is collected - goes back to 30 MB. Everything works correct, it is just not efficient way to do that. Sadly, there is no video.getImageData() :( – Ivan Kuckir Sep 21 '12 at 09:58
2

clearRect() is what I think you are looking for.

To the best of my knowledge, uploading images on canvases takes space, but clearRect() deletes all memory on the canvas.

What I did on JS code:

  1. On the Code down below is I set a stream on a video tag. See https://www.kirupa.com/html5/accessing_your_webcam_in_html5.htm for more WebCam info.

  2. I set an interval of one second(calls function every second) to Clear the canvas and then draw what the stream was on that frame.

  3. Now It your turn. Minimizing the video or canvas should have no effect on what to do. now you can grab the pixle colors on the webcam or video whenever.

Here is the Working Code:

Note: I found that this Example does not work with the built in web viewer. This example needs it's own web page.

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta content="stuff, to, help, search, engines, not" name="keywords">
<meta content="What this page is about." name="description">
<meta content="Display Webcam Stream" name="title">
<title>Display Webcam Stream</title>
  
<style>
#container {
    margin: 0px auto;
    width: 500px;
    height: 375px;
    border: 10px #333 solid;
}
#videoElement {
    width: 500px;
    height: 375px;
    background-color: #666;
}
#canvas {
  background-color: #666;
}
</style>
</head>
  
<body>
    <video autoplay="true" id="videoElement"></video>
    <canvas id="canvas" onclick= canvasOnclick()></canvas>

<script>
var video = document.querySelector("#videoElement");
var canvas = document.getElementById('canvas');
var ctx = canvas.getContext('2d');
if (navigator.mediaDevices.getUserMedia) {       
    navigator.mediaDevices.getUserMedia({video: true})
  .then(function(stream) {
    video.srcObject = stream;
  })
  .catch(function(err0r) {
    console.log("Something went wrong!");
  });
}



var myVar = setInterval(intervalTimer, 1000);

function intervalTimer() {
  ctx.clearRect(0,0,400,400);
  ctx.drawImage(video,0,0,400,400);

}

//var data = ctx.getImageData(0, 0, w, h).data;
</script>
</body>
</html>
TripletForce
  • 21
  • 2
  • 1
    Thanks! But I don't want to draw a video to Canvas. I want to get pixel data (values of Red, Green and Blue of each pixel of the current video frame), so I can loop through pixels with my code, compute an average color, etc. – Ivan Kuckir Jun 22 '18 at 05:12
0

canvas{display:none} may help reducing some overhead. Canvas remains accessible to js.

joox
  • 243
  • 5
  • 13