In webrtc is it possible to get video frames without using canvas?

Question

My actual project is to develop a video call application using webrtc and transfer the video audio frames for NDI SDK, NDI input should be in frames. For this now I have done this using canvas but the delay is high. If frames can be obtained directly from the camera directly or have any other method to convert streams to frames?

This is the sample code how I get the frames now (var frame = this.ctx1.getImageData(0, 0, this.width, this.height);)

var processor = {
  timerCallback: function() {
    if (this.video.paused || this.video.ended) {
      return;
    }
    this.computeFrame();
    var self = this;
    setTimeout(function () {
      self.timerCallback();
    }, 16); // roughly 60 frames per second
  },

  doLoad: function() {
    this.video = document.getElementById("my-video");
    this.c1 = document.getElementById("my-canvas");
    this.ctx1 = this.c1.getContext("2d");
    var self = this;

    this.video.addEventListener("play", function() {
      self.width = self.video.width;
      self.height = self.video.height;
      self.timerCallback();
    }, false);
  },

  computeFrame: function() {
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
    var frame = this.ctx1.getImageData(0, 0, this.width, this.height);
    var l = frame.data.length / 4;

    for (var i = 0; i < l; i++) {
      var grey = (frame.data[i * 4 + 0] + frame.data[i * 4 + 1] + frame.data[i * 4 + 2]) / 3;

      frame.data[i * 4 + 0] = grey;
      frame.data[i * 4 + 1] = grey;
      frame.data[i * 4 + 2] = grey;
    }
    this.ctx1.putImageData(frame, 0, 0);

    return;
  }
};

Is this video from a local camera? Or, the remote end? How are you communicating with the NDI SDK? In any case, consider using `requestAnimationFrame()` instead of your timer. If you're making your own browser wrapper, you can probably hook whatever is doing the drawing of the video and send that data straight off to the NDI SDK. I've seen similar code for this around some years back, but can't find it now. I think a signage or scoreboard company had open sourced their build of Chromium.... — Brad, Jun 22 '21 at 06:18
Local Camera itself. From the local camera to have to get the frames, depend on this process I have to decide how to transfer frames to NDI either as a Remote user or socket itself. Thanks @Brad I am checking signafe and scoreboard now. — arun n a, Jun 22 '21 at 06:22
What format do you need? [Possible duplicate?](https://stackoverflow.com/questions/66925427/get-readablestream-from-webcam-in-browser/66927198#66927198) — Kaiido, Jun 22 '21 at 06:41
@arunna This is what I was thinking of: https://github.com/daktronics/cef-mixer Not quite what you're looking for, but something to investigate if you want to dump the whole thing. Realistically though, why not just connect the camera to NDI directly? Perhaps via GStreamer or something if you need something special. — Brad, Jun 22 '21 at 06:48
So WebCodecs might indeed be a solution, but not knowing NDI myself, I think you should rather consider Brad's suggestions first. — Kaiido, Jun 22 '21 at 06:51
@Brad let me check the git. No we couldn't connect the camera directly to NDI because here we had room concept and a moderator will be switching the presenters from one room to another and in each room, there will be multiple audience too. — arun n a, Jun 22 '21 at 06:52

score 1 · Answer 1 · answered Jun 22 '21 at 16:02

I don't think there is another method to access (and edit) video frames in live.

But as suggested in this answer, rather than using requestAnimationFrame function, you can use (on chrome only) requestVideoFrameCallback API. As said here :

The requestVideoFrameCallback() method allows web authors to register a callback that runs in the rendering steps when a new video frame is sent to the compositor. This is intended to allow developers to perform efficient per-video-frame operations on video, such as video processing and painting to a canvas, video analysis, [...]

On the other hand, as say in this post, canvas operations are heavy on CPU. To simply edit a video display, you should rather try to use some CSS3 transform.

mjpolak · Answer 2 · 2021-06-22T12:16:45.470

-1

Instead of assigning MediaStream to canvas, you may create ImageCapture object and use takePhoto Method, see

https://developer.mozilla.org/en-US/docs/Web/API/ImageCapture

for Audio you may use: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder see requestData method and ondataavailable

edited Jun 22 '21 at 12:16

answered Jun 22 '21 at 06:35

mjpolak

721
6
24

In this case we won't get the audio right? – arun n a Jun 22 '21 at 06:47
1

This API is not appropriate for a stream. It's designed for web pages emulating cameras. Photo capture works differently, and applies different constraints for taking the photo. – Brad Jun 22 '21 at 06:49
@brad Why u have such information, you may check the example from the link above here: https://googlechrome.github.io/samples/image-capture/grab-frame-take-photo.html and it is working as expected. – mjpolak Jun 22 '21 at 07:24
@mjpolak I checked the example does this line (imageCapture.grabFrame()) gives frames? – arun n a Jun 22 '21 at 08:58
Check my answer, I don't suggest using `grabFrame` but `takePhoto` which returns `Blob` with image data. – mjpolak Jun 22 '21 at 09:02
@mjpolak what should i do to get the audio frames – arun n a Jun 22 '21 at 11:07
I edited post about `MediaRecorder` which may be used for grabbing audio. – mjpolak Jun 22 '21 at 12:17
I am already using MediaRecorder but its delay is high, so cannot be used in live communication. – arun n a Jun 22 '21 at 15:49

In webrtc is it possible to get video frames without using canvas?

2 Answers2