First frame from captureStream() not sending

Question

We're working on a project where people can be in a chat room with their webcams, and they can grab a snapshot of someone's cam at that moment, do some annotations on top of it, and then share that modified picture as if it was their own webcam (like sharing a whiteboard).

Capturing the webcam stream into a canvas element where it can be edited was relatively easy. Finding the canvas element on our page and doing a .getContext('2d') on it, Used an open library to add editing tools to it. Grabbing a stream from that canvas was done like so:

var canvasToSend = document.querySelector('canvas');
var stream = canvasToSend.captureStream(60);
var room = osTwilioVideoWeb.getConnectedRoom();

var mytrack = null;
room.localParticipant.publishTrack(stream.getTracks()[0]).then((publication) => {
    mytrack = publication.track;
    var videoElement = mytrack.attach();
});

This publishes the stream alright, but the first frame will not get sent unless you draw something else on the canvas. Let's say you drew 2 circles and then hit Share, the stream will start but will not be shown on the recipients' side unless you draw a line, or another circle, or anything. It seems like it needs a frame change for it to send data over.

I was able to force this with developer tools by doing something like context.fill();, but when I tried adding this after the publishing function, even in a then()... no luck.

Any ideas on how to force this "refresh" to happen?

That all sounds like a Chrome bug. I don't have enough time tonight to dig into it, but even [requestFrame](https://developer.mozilla.org/en-US/docs/Web/API/CanvasCaptureMediaStreamTrack/requestFrame) which should have done just what you want doesn't seem to work... Maybe the best workaround for now would be `ctx.drawImage(canvas);`once the stream is being consumed. — Kaiido, Jun 19 '18 at 15:23
That's crossed my mind, but the result is the same in Firefox unfortunately. Will try that workaround, I'm just having trouble with the whole "once the stream is being consumed", I haven't found a reliable way to run those functions *only after* the stream begins. I have a feeling they're running before. — marianopicco, Jun 19 '18 at 16:33
Uh? Can you confirm [this fiddle](https://jsfiddle.net/ta0f6g9d/) reproduces the issue on your Firefox? If so, does clicking "requestFrame" draw the frame there? I cannot repro on my FF62 and FF61 on macOs. If it doesn't there may be something in your server side too. I don't know twilio but fast reading the Internets, there seem to be a `participant.on('trackPublished')` and a `participant.on('trackAdded')` event listeners. That's probably where you should refresh the frame (and once again from my tests on Chrome, `gCO='copy'; ctx.drawImage(ctx.canvas,0,0)` seems to be the cleanest on Chrome. — Kaiido, Jun 20 '18 at 01:13

score 5 · Answer 1 · answered Jun 20 '18 at 14:39

So it seems it is expected behavior (and thus would make my FF buggy).

From the specs about the frame request algorithm:

A new frame is requested from the canvas when frameCaptureRequested is true and the canvas is painted.

Let's put some emphasis on the "and the canvas as been painted". This means that we need both these conditions, and while captureStream itself, or its frameRate argument ellapsing or a method like requestFrame would all set the frameCaptureRequested flag to true, we still need the new painting...

The specs even have a note stating

This algorithm results in a captured track not starting until something changes in the canvas.

And Chrome indeed seems to generate an empty CanvasCaptureMediaStreamTrack if the call to captureStream has been made after the canvas has been painted.

const ctx = document.createElement('canvas')
  .getContext('2d');
ctx.fillRect(0,0,20,20);
// let's request a stream from before it gets painted
// (in the same frame)
const stream1 = ctx.canvas.captureStream();
vid1.srcObject = stream1;
// now let's wait that a frame ellapsed
// (rAF fires before next painting, so we need 2 of them)
requestAnimationFrame(()=>
  requestAnimationFrame(()=> {
    const stream2 = ctx.canvas.captureStream();
    vid2.srcObject = stream1;
  })
);

<p>stream initialised in the same frame as the drawings (i.e before paiting).</p>
<video id="vid1" controls autoplay></video>
<p>stream initialised after paiting.</p>
<video id="vid2" controls autoplay></video>

So to workaround this, you should be able to get a stream with a frame by requesting the stream from the same operation as a first drawing on the canvas, like stream1 in above example.

Or, you could redraw the canvas context over itself (assuming it is a 2d context) by calling ctx.drawImage(ctx.canvas,0,0) after having set its globalCompositeOperation to 'copy' to avoid transparency issues.

const ctx = document.createElement('canvas')
  .getContext('2d');
ctx.font = '15px sans-serif';
ctx.fillText('if forced to redraw it should work', 20, 20);
// produce a silent stream again
requestAnimationFrame(() =>
  requestAnimationFrame(() => {
    const stream = ctx.canvas.captureStream();
    forcePainting(stream);
    vid.srcObject = stream;
  })
);
// beware will work only for canvas intialised with a 2D context
function forcePainting(stream) {
  const ctx = (stream.getVideoTracks()[0].canvas ||
      stream.canvas) // FF has it wrong...
    .getContext('2d');
  const gCO = ctx.globalCompositeOperation;
  ctx.globalCompositeOperation = 'copy';
  ctx.drawImage(ctx.canvas, 0, 0);
  ctx.globalCompositeOperation = gCO;
}

<video id="vid" controls autoplay></video>

First frame from captureStream() not sending

1 Answers1