Unable jump into stream from media recorder using media source with socket.io

Question

The following code works when the video observing client is loaded first then the webcam client is loaded second it works flawlessly, however, if the order is switched or in any way the stream is interrupted for example by refreshing either client the stream will fail and the Media Source will change its ready state to closed.

My assumption is that the video being received on start needs initialization headers for starting and since the stream is being read midstream it never gets said initialization headers. I am unsure of how to even add such header to the webm file.

I have tried to change the sequence mode on the source buffer which did nothing. I have tried restarting the video recorder and that works, but my final plan is to have multiple observing clients and the video recorder restarting on every reconnection is not optimal.

Camera Client

main();
function main() {
    if (hasGetUserMedia()) {
        const constraints = {
            video: {
                facingMode: 'environment',
                frameRate: {
                    ideal: 10,
                    max: 15
                }
            },
            audio: true
        };

        navigator.mediaDevices.getUserMedia(constraints).
        then(stream => {
            setupRecorder(stream);
        });
    }
}

function setupRecorder(stream) {
    let mediaRecorder = new MediaRecorder(stream, {
        mimeType: 'video/webm; codecs="opus, vp9"'
    });

    mediaRecorder.ondataavailable = e => {
        var blob = e.data;
        socket.emit('video', blob);
    }

    mediaRecorder.start(500);
}

The server just broadcasts whatever is received

Observing Client

var sourceBuffer;
var queue = [];
var mediaSource = new MediaSource();
mediaSource.addEventListener('sourceopen', sourceOpen, false);
main();

socket.on('stream', data => {
    if (mediaSource.readyState == "open") {
        if (sourceBuffer.updating || queue.length > 0) {
            queue.push(data.video);
        } else {
            sourceBuffer.appendBuffer(data.video);
        }
    }
});

function main() {
    videoElement = document.querySelector('#video');
    videoElement.src = URL.createObjectURL(mediaSource);
}

function sourceOpen(e) {
    console.log('open');
    sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs="opus, vp9"');
    sourceBuffer.addEventListener('updateend', () => {
        console.log(sourceBuffer.updating, mediaSource.readyState);

        if (queue.length > 0 && !sourceBuffer.updating) {
            sourceBuffer.appendBuffer(queue.shift());
        }
    });
}

So the code, in fact, works just in a way that is not correct so nothing is wrong with the server of socket sending. It either has something to do with the MediaRecorder or MediaSource.

Answer 1

My assumption is that the video being received on start needs initialization headers for starting and since the stream is being read midstream it never gets said initialization headers.

Correct!

To solve this, you need to know a bit about the WebM format. WebM is just a subset of Matroska (MKV). Matroska is a specification of a schema for storing media in EBML. EBML is a binary file format that can have arbitrary blocks. Think of it like a binary XML.

What this means is that you can use tools like EBML Viewer to inspect WebM files, and reference the Matroska specs to understand what's going on. For example:

This is the inspection of a WebM file that was pre-recorded. It will play fine in browsers. You'll note that there are elements that are nested.

There are two top-level elements found in every WebM file. EBML, which defines this binary file, and Segment which contains everything after.

Within Segment there are a couple elements that matter to you. One of which is Tracks. You'll note that this file has two tracks, one for audio in Opus, and one for video in VP9. The other important block is Info, which contains information about the timescale and some metadata about the muxer.

After all of that metadata, you find Cluster, Cluster, Cluster, etc. These are the places in which you can cut a WebM stream, provided that each Cluster begins with a keyframe.

In other words, your code should do the following:

• Save all data before the first Cluster as "initialization data".
• Split on Cluster after that.

On playback:

• Use the previously-saved "initialization data" as the first thing you load in.
• Start loading in Clusters after that, starting wherever you want in the stream.

Now, that who cluster-needs-a-keyframe bit is important. As far as I know, there's no way to configure MediaRecorder to do this, and browsers are particularly picky about this. At a minimum, you'll have to remux server-side... you might even need to re-encode. See also: Encoding FFMPEG to MPEG-DASH – or WebM with Keyframe Clusters – for MediaSource API

using media source with socket.io

I should point out that you don't even need MediaSource for this. You definitely don't need Socket.IO. It can be as simple as outputting this data over a normal HTTP stream. This is loadable directly in a <video> element. (By all means, use MediaSource if you want additional control, but it isn't necessary.)