0

I hope to be as properly concise as possible.

The process:

First I start by send chunks from a stream via MediaRecorder and socket.io to my node server. The server then appends the blob.data that was sent to a static variable. Then a user starts a video stream with a given url to my node server which returns a buffer of video/mp4 data.

The code:

Client sends stream recording via MediaRecorder and socket.io:

  successCallback(stream) {
    if (MediaRecorder.isTypeSupported('video/mp4')) {
      this.options = { mimeType: 'video/mp4' };
    }

    this.video.srcObject = stream;
    this.video.play();
    this.mediaRecorder = new MediaRecorder(stream, this.options);
    this.mediaRecorder.ondataavailable = this.handleDataAvailable.bind(this);
    this.mediaRecorder.start(3000);
  }

  handleDataAvailable(blob) {
    // POST/PUT "Blob" using FormData/XHR2
    this.signalHub.sendStream(blob.data);
  }

  public sendStream(stream) {
    this.socket.emit('send-stream', stream);
  }

Server receives the data:

socket.on('send-stream', (newStreamBuff) => {
  //stream = Buffer.concat([headerStream, newStreamBuff]);
  stream = newStreamBuff;
  console.log(stream);
  console.log(stream[0]);
});

Client starts the stream

  startExternalVideo() {
    this.recordedVideo.srcObject = 'http://ec2-xx-xx-xxx-xxx.compute-1.amazonaws.com:4040/video';

    let promise = this.recordedVideo.play();
    if (promise !== undefined) {
      promise.then(datas => {
        console.log(datas);
      }).catch(error => {
        console.log(error);
      });
    }
  }

Server gets the request that the user started the stream:

app.get('/video', function(request, response){
  const head = {
    'Content-Length': stream.byteLength,
    'Content-Type': 'video/mp4',
  }
  response.writeHead(200, head);
  response.write(stream, 'binary');
  response.end(null, 'binary');
});

This sort of works. If it stops recording before the second buffer is sent to the server. I can go to the link provided to get the stream and see the first buffer (It's only two seconds long). But anything else after that, I can't view it. The stream is also not continuous (viewing one buffer to another (which seems like another problem)).

Now here is what I'm trying to fix. It seems like each buffer after the first buffer sent doesn't send the file signature. Notice:

<Buffer 1a 45 df a3 a3 42 86 81 01 42 f7 81 01 42 f2 81 04 42 f3 81 08 42 82 88 6d 61 74 72 6f 73 6b 61 42 87 81 04 42 85 81 02 18 53 80 67 01 ff ff ff ff ff ... >
<Buffer 8c 81 0b 40 80 fb 03 ff fe ff fe ff fe a3 8c 81 0b 7c 80 fb 03 ff fe ff fe ff fe a3 45 51 82 0b 7c 00 00 00 00 01 09 30 00 00 00 01 61 9a 57 8b d5 ca ... >
<Buffer 8c 81 07 80 80 fb 03 ff fe ff fe ff fe a3 42 43 82 07 80 00 00 00 00 01 09 30 00 00 00 01 61 9a 3b 84 10 27 82 13 f6 77 9c 4f 0c 8a e3 85 70 d8 a7 c5 ... >

Notice the first four hexadecimals in the buffer.

1a 45 df a3

That is the webM file signature. I think because the other buffers don't have that file signature is the reason why I can't or view the other buffers when my node server gets updated.

JD333
  • 501
  • 8
  • 22

1 Answers1

0

It seems like each buffer after the first buffer sent doesn't send the file signature.

It's not supposed to. It's up to you to segment and reassemble the stream.

You can treat everything in the stream up to the first Cluster element as initialization data. After that, you can drop into the stream by starting at an arbitrary Cluster element, as long as that Cluster starts with a keyframe. (Unfortunately, Media Recorder doesn't guarantee this, and has no options for setting the keyframe interval.)

See also: https://stackoverflow.com/a/56062582/362536

Brad
  • 159,648
  • 54
  • 349
  • 530
  • This is helpful. I was looking at your post. Can you recommend anything else besides WebM. Or do you think this is a good choice? Looks like I got a lot of work – JD333 Aug 03 '19 at 21:37
  • WebM is the way to go, in my strong opinion. You basically have two choices... WebM or MP4. WebM is straightforward, simple, free from licensing/patent cruft, and is well supported. Of course, MP4 is also in heavy use and is well supported. – Brad Aug 03 '19 at 21:44
  • So I recorded the stream as mp4. Why is the file signature WebM. Just curious – JD333 Aug 03 '19 at 21:54
  • When you set up the MediaRecorder, you can only suggest what file format to record into. `video/mp4` must not have been supported on your platform. Chrome on Windows returns `false` for `MediaRecorder.isTypeSupported('video/mp4')` for me. – Brad Aug 03 '19 at 22:14
  • Ah gotcha weird. Yes, I was using chrome on windows. – JD333 Aug 03 '19 at 22:25
  • It probably has to do with H.264 licensing. You're probably getting VP8 video, and I don't think the MP4 container supports it, whereas WebM does. – Brad Aug 03 '19 at 22:34
  • as each buffer gets sent to my server. That's where I will create the clusters. It's really only clusters I need correct? So can I consider each buffer to be a cluster? – JD333 Aug 03 '19 at 23:18
  • What I'm referring to "cluster" in this context is a Matroska/WebM cluster. (WebM is a subset of Matroska... uses the exact same structure.) Check this diagram: https://www.matroska.org/technical/diagram/index.html The Cluster element (and everything else) is generated from the MediaRecorder. I don't think MediaRecorder guarantees that the chunks it emits will be on any particular boundary... you'll have to segment this yourself. – Brad Aug 03 '19 at 23:23
  • You might find this package useful: https://www.npmjs.com/package/ebml – Brad Aug 03 '19 at 23:24
  • So looking at the diagram.. is all in the blob i send to the server. Or do you mean only the cluster element and everything else inside of it is generated by the media recorder? – JD333 Aug 03 '19 at 23:35
  • What do you mean by particular boundary.. sorry. This really helpful to me – JD333 Aug 03 '19 at 23:35
  • Yes, if you concatenate all of those blobs, you'll get this whole WebM/MKV structure, complete with cluster elements. And, what you'll need to do is look for the start of clusters (they start with `0x1F43B675`). So, all of the data up to the first cluster can be considered "initialization data". And, after that, you can pick up streaming on any cluster... **as long as that cluster starts with a keyframe**, which isn't guaranteed by MediaRecorder. – Brad Aug 03 '19 at 23:39
  • So in the snippet of buffer data I showed you. The first one is probably the header, Meat Seek Info, Segment Info, Track, Chapters, and the other two are the cluster? – JD333 Aug 04 '19 at 01:18
  • I was thinking since it's a stream of data. A video that's being streamed. I wouldn't necessarily want to keep all of previous parts of the stream correct? – JD333 Aug 04 '19 at 01:22
  • So when parsing the blob.data server side: When I reach the hexadecimal (0x1F43B675) I reached a cluster. Everything before that I should save as initialization data. Everything after that will be my cluster. Every proceeding blob.data after that doesn't start with a cluster or (0x1F43B675) I'm not sure what it starts with. Where as the first blob.data starts with I'm guessing the EBML, Info, Tracks, and tags correct? Do the proceeding blobs start with only the Info, Tracks, and tags? – JD333 Aug 05 '19 at 17:20
  • Uh so brad, I downloaded the EBML Viewer 2.0. When using it for the first buffer that gets passed to my server. I get the variable-length integer 0x01FFFFFFFFFFFFFF does not represent valid entry size (file position #44) and then when I try it with multiple buffers concatenated to one buffer I get this error. com.google.code.ebmlviewer.core.EbmlFormatException: length descriptor is 0b00000000 (file position #0) – JD333 Aug 07 '19 at 01:02
  • Is the above problem an issue because I'm not parsing the WebM file on the server side properly? Or at all? – JD333 Aug 07 '19 at 01:04
  • Since the MediaRecorder is outputting a stream, it uses that indeterminate length in the header. EBML Viewer can't handle this, sadly. Try opening up a normally recorded WebM with EBML Viewer, and you'll see what I mean. – Brad Aug 07 '19 at 01:33
  • @JD333 There's a lot of questions here. :-) If you're interested, I'm available for consulting... I think I can probably get you pointed in the right direction in half hour or an hour. Shoot me an e-mail at brad@audiopump.co if you're interested, thanks. I think that might be faster to get you on-track for your project. – Brad Aug 07 '19 at 01:34
  • Yeah - finally had a chance to think a bit more about it. Hah, depends on what you're charging! And I'm checking out the ebml package. Seems to be a little outdated? Instantiating Decoder gives me an error that Decoder is not a constructor (which I see in the ebml.js) Still learning the difference between js and node I guess. – JD333 Aug 07 '19 at 01:48