9

I've been digging through the Media Source Extension examples on the internet and haven't quite figured out a way to adapt them to my needs.

I'm looking to take a locally cached MP4/WebM video (w/ 100% keyframes and 1:1 ratio of clusters/atoms to keyframes) and decode/display them non-sequentially (ie. frame 10, 400, 2, 100, etc.) and to be able to render these non-sequential frames on demand at rates from 0-60fps. The simple non-MSE approach using the currentTime property fails due to the latency in setting this property and getting a frame displayed.

I realize this is totally outside normal expectations for video playback, but my application requires this type of non-sequential high speed playback. Ideally I can do this with h264 for GPU acceleration but I realize there could be some platform specific GPU buffers to contend with, though it seems that a zero frame buffer should be possible (see here). I am hoping that MSE can accomplish this non-sequential high framerate low latency playback, but I know I'm asking for a lot.

Questions:

  1. Will appendBuffer accept a single WebM cluster / MP4 Atom made up of a single keyframe, and also be able to decode at a high frequency (60fps)?

  2. Do you think what I'm trying to do is possible in the browser?

Any help, insight, or code suggestions/examples would be much appreciated.

Thanks!

Update 4/5/16

I was able to get MSE mostly working with single frame MP4 fragments in Firefox, Edge, and Chrome. However, Chrome seems to be running into the frame buffer issue linked above and I haven't found a way to pre-process a MP4 to invoke this "low delay" mode. Anyone have any clues if it's possible to create such a file with an existing tool like MP4Box?

Firefox and Edge decode/display the individual frames immediately with very little latency, but of course something breaks once I load this video into a Three.js WebGL project (no video output, no errors). I'm ignoring this for now as I'd much rather have things working on Chrome as I'll be targeting Android as well.

Dustin Kerstein
  • 450
  • 5
  • 13
  • Regarding currentTime: did you try it with the "seeked" event? If not, this might work for you: http://stackoverflow.com/questions/19175174/capture-frames-from-video-with-html5-and-javascript/19176124#19176124 –  Apr 02 '16 at 01:23
  • Never tried it, and I think only latest Chrome versions implement it, but I think you can use the segment appendMode on SourceBuffer. See https://www.w3.org/TR/media-source/#idl-def-AppendMode.sequence. – Pablo Montilla Apr 04 '16 at 12:52
  • @K3N I hadn't actually. It definitely makes things a bit more consistent as it's not trying to update currentTime when it's still seeking (which may introduce further delays). I setup a quick test on my Surface Pro 2 with a 1080p h264 MP4 incrementing currentTime as fast as possible using the "seeked" event. Results - 30fps on Firefox, 60fps on Microsoft Edge, but Chrome lags far behind at 6fps. I'll try to test Chrome on Android to see if it's just as bad. – Dustin Kerstein Apr 04 '16 at 16:07
  • @PabloMontilla sourceBuffer.mode = 'sequence' certainly sounds like something I'd need to use for an MSE implementation. Thanks for the suggestion. – Dustin Kerstein Apr 04 '16 at 16:29
  • 1
    @K3N I figured out why Chrome was performing so poorly in my test above. It seems that there is a bug in Chrome that causes it to fallback to software decode when the resolution > 1080p - https://bugs.chromium.org/p/chromium/issues/detail?id=573565&q=h264%20software%20decoder&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified – Dustin Kerstein Apr 11 '16 at 15:25

1 Answers1

8

I was able to get this working pretty well. The key was getting Chrome to enter its "low delay" mode by muxing a specially crafted MP4 file using modified mp4box sources. I added one line in movie_fragments.c so it read:

        if (movie->moov->mvex->mehd && movie->moov->mvex->mehd->fragment_duration) {
            trex->track->Header->duration = 0;
            Media_SetDuration(trex->track);
            movie->moov->mvex->mehd->fragment_duration = 0;
        }

Now every MP4 created will have the MEHD fragment duration set to 0 which causes Chrome to process it as a live stream.

I still have one remaining issue related to the timestampOffset property which in combination with the FPS set in the media fragments control the playback speed. Since I'm looking to control the FPS directly I don't want any added delay from the MSE playback engine. I'll post a separate question here to address that.

Thanks, Dustin

Dustin Kerstein
  • 450
  • 5
  • 13