0

I am currently encoding live video and streaming to the browser. I encode rgb->vuy->h264->Wrap in (MP4 ISO BMFF Byte Stream Format spec, section 3)-> websocket to client. first package is initialization data from ftyp to moov tag.

And from there on I am sending moof and mdat tags to the client. And I stream the video.

No matter how many frames are in mdat chunk. The MSE always buffers the data to have seamless play. How can one overwrite this using MSE.

I read somewhere that there is a way mdat understands that it is a live steaming webm chunk. But I do not know if such think exist for mp4 or exists altogether.

Evren Bingøl
  • 1,306
  • 1
  • 20
  • 32
  • I found this, https://stackoverflow.com/questions/29636740/make-an-html5-video-buffer-less-before-playing but there is not a real answer – Evren Bingøl Jun 20 '19 at 18:53
  • How are you creating the mp4, and how are you controlling the chunk size? – szatmary Jun 20 '19 at 21:16
  • i use Microsoft sinker(Fragmented version), Then loop and change some values in the mp4 Tags such as tfdt trun and stuff to make it Spec 3 compliant so that MSE wont complain. I have full control over the encoder. So The more frames I put to the sinker the more chunks I get. But the issue is on the client side. even if I write 2 frames per chunk(mdat) there is still a latency as MSE need to buffer before start rendering. – Evren Bingøl Jun 20 '19 at 23:42
  • I have never hear of Microsoft sinker. And google returns no results. Are you getting one moof every two frames? Or just one mdat every two frames? Can you post a sample file? – szatmary Jun 20 '19 at 23:45

1 Answers1

1

I'm sending moof once per one frame and it works.

Max Lapshin
  • 822
  • 7
  • 15
  • Yeah, in my experience moof for each frame is the best way to reduce latency. I actually checked Chrome sources and they also tread moofs as fragments that should be completely received to start decoding. – Rudolfs Bundulis Jun 21 '19 at 11:20
  • I send ftyp and moov initially. Then I send (moof and mdat) . mdat might have one or more frames. Is that right? I am getting about 1 or 2s delay on local network. I am trying to get to 0.2/0.3s. I send about 20 fps. If i get up to 30 FPS, tcp buffers eventually fill up. My encode is fast, I encode with Intel or Nvidia harware encoders. So I can get up to 60 fps easy on encode side. I am streaming my desktop so I can see the cursor being late. screen resolution is 1080x1920 So frame size matters too. What resolution are you guys sending? – Evren Bingøl Jun 21 '19 at 16:52
  • A related question: https://stackoverflow.com/questions/54186634/sending-periodic-metadata-in-fragmented-live-mp4-stream – El Sampsa Dec 11 '19 at 16:00