13

I am aware of how to retrieve eg. a video stream from a webcam source:

const constraints = { video: true };

navigator.mediaDevices

    .getUserMedia(constraints)

    .then(mediaStream => {

        // ** But how to get the a stream of bytes from here??? **

    });

I could not find any proper documentation of how to retrieve a stream of bytes from the mediaStream object.

How to do this? Because, lets say I want to stream the bytes to the server.

Dachstein
  • 3,994
  • 6
  • 34
  • 61
  • I was exactly looking for this. Have you experimented with this more? e.g. sending the audio/video bytes to the server and then transmitting to the other device. So that it works like VoIP (similar to WebRTC)? We are looking to implement the VoIP using our custom server, possibly without WebRTC. – iammilind Mar 26 '19 at 14:32
  • @iammilind Not yet, I was just down the standard WebRTC path. Post your own experience in the future here if you made some interesting discovering. – Dachstein Mar 27 '19 at 08:35

1 Answers1

10

MediaStream Recording API

By further investigating into MDN and the HTML 5 APIs related to Audio and Video I have found the MediaStream Recording API.

So, to get the byte stream (or chunks as soon as some are available) we can do this:

const constraints = { video: true };

navigator.mediaDevices

    .getUserMedia(constraints)

    .then(mediaStream => {

        // use MediaStream Recording API
        const recorder = new MediaRecorder(stream);

        // fires every one second and passes an BlobEvent
        recorder.ondataavailable = event => {

            // get the Blob from the event
            const blob = event.data;

            // and send that blob to the server...
        };

        // make data available event fire every one second
        recorder.start(1000);
    });

Browser Support:

The MediaStream Recording API is still in Working Draft status (March 2018). Currently only natively supported in Chrome and Firefox.

Polyfill: streamproc/MediaStreamRecorder

Further reading: Record to an audio file

Dachstein
  • 3,994
  • 6
  • 34
  • 61
  • 3
    In my experience it seems like the MediaStream Recording API doesn't actually give you the raw bytes from the MediaStream, but actually **re-encodes** the video. I noticed that when a video is sent over WebRTC and you use the MediaStream Recording API to read the raw bytes that the encoding of the video changes, the PTS timestamps are all rebased to 0, and there is a 1-2 second delay between the original video and the new video (likely a buffer for bi-directional frames) – stevendesu Dec 05 '18 at 00:09
  • Indeed, the question was asking how to access the raw stream of bytes on the mediastream / mediastreamtrack itself. So even though the solution mentioned work, its doesn't answer the question. Also, its not cross browser compatible a MediaRecorded does not have good support. Any hints on accessing the direct stream of bytes on the MediaStreamTrack? – Lou-adrien Oct 06 '20 at 13:25
  • 1
    How about drawing the video on a `Canvas` with a `RequestAnimationFrame` loop and then you can access every pixel. What that solve your problem? Need some code, let me know. – Dachstein Oct 16 '20 at 14:59