webrtc captureStream from canvas and capture audio from video

Question

In my project I have a video displayed. On top of the video there is a canvas where some object are drawn. Therefore the video is overlayed with some drawings. Now I want to download the video file including the drawings. I am using the webrtc mediaRecorder. I am drawing the video to a canvas on window.requestAnimationFrame. My exported video looks good but there is no audio from the video because I captured the canvas. Is it possible to capture the audio from the video and add it to the canvasStream? I also recorded the videostream including the audio but then I dont have my drawings on top of it. Any suggestions?

Answer 1

You need to create new MediaStream with input stream AudioTrack and Canvas Video Track.
Then record the new Stream, so the recorder output(blobs) will contains both audio and video.

var options = {mimeType: 'video/webm'};
var recordedBlobs = [];
var newStream = new MediaStream();
newStream.addTrack(inputStream.getAudioTracks()[0]);
newStream.addTrack(canvasStream.getVideoTracks()[0]);
mediaRecorder = new MediaRecorder(newStream, options);
mediaRecorder.ondataavailable = function (event) {
    if (event.data && event.data.size > 0) {
        recordedBlobs.push(event.data);
    }
}
mediaRecorder.start(1000); 

See my demo.