1

When the user does not have a video device, I'm using the canvas video stream and merging the video track with the stream obtained from getUserMedia(audio stream). Peer connects perfectly but other users are unable to hear the audio.

If I share my screen and merge this video track into the stream, then the audio works perfectly. I noticed that CanvasMediaStream is not working but MediaStream is working perfectly. I don't know if this is an issue at kurento's end or if I am doing something wrong.

Shubham
  • 150
  • 1
  • 7
  • CanvasMediaStream shouldn't exist anymore... Does this also happen on Chrome? Can you hear the audio when consumed locally? What do you want from us exactly? Sounds like you already know the fix: add the canvascapturemediastreamtrack to the audio MediaStream or even create a new MediaStream from both tracks. – Kaiido Jan 21 '21 at 23:52
  • Thanks for the reply @Kaiido . Without video streaming, kurento is not working, because of this I tried to add dummy video track. With this, the call initiates but the audio is not audible. This happens on both Chrome and Firefox browsers. Is there any other way to add video track ? I can hear the audio if I add a screen sharing video track. I just want other users to hear my voice in all aspects. – Shubham Jan 22 '21 at 07:32
  • 1
    If it also happens on Chrome then it's not because the stream is a CanvasMediaStream. Still you could try to do `new MediaStream([ canvasStream.getVideoTracks()[0], micStream.getAudioTracks()[0] ])`. Also you may want to call some drawing methods at regular interval (at least every 5s on Chrome) on the canvas's context on which you grabbed the stream, even `ctx.clearRect()` should do, but to be safe start with `fillRect(0,0,1,1)`. Failing to do so, the browser may mute the canvas track, which may be an issue for kurento (that I don't know). – Kaiido Jan 22 '21 at 07:42
  • @Kaiido okay thanks, will try this solution. But apart from using the canvas stream, do I have any alternate options? – Shubham Jan 22 '21 at 07:45
  • 1
    If you need a video stream? Not sure... Even a short looping MediaElement's MediaStream will emit the mute event at every loop, so if your backend really have issues with muted tracks, it may also fail, though might be worth the try too, but I suspect just drawing on the canvas context every 0.5 seconds will do. – Kaiido Jan 22 '21 at 07:59

1 Answers1

3

No Kurento to test, but the problem is probably that the CanvasCaptureMediaStreamTrack gets muted after some time of inactivity on the canvas's context.

To workaround that you can simply set up a drawing loop that will update the canvas regularly (every half seconds should be largely enough without causing too much overhead either).

Also, you may want to start from a fresh MediaStream, though I doubt this has any influence:

// assumes there is a 'canvas' and a 'mic_stream'

// make the context active, so the stream is not muted
const ctx = canvas.getContext("2d");
setInterval(() => ctx.clearRect(0,0,1,1), 500);
ctx.clearRect(0,0,1,1);
const canvas_stream = canvas.captureStream();

const canvas_track = canvas_stream.getVideoTracks()[0];
const mic_track = mic_stream.getAudioTracks()[0];
const merged_stream = new MediaStream([ canvas_track, mic_track ]);

// do something with 'merged_stream'
Kaiido
  • 123,334
  • 13
  • 219
  • 285