3

I want to make a web app, which extracts each frame of a MP4 file, and saves it as a PNG. Setting the currentTime of a <video> element and drawing it into a canvas does not work, as it often skips some frames.

I am using the VideoDecoder. It has a decode method, which receives an EncodedVideoChunk, that contains an array of bytes - a "chunk of a video".

I parsed this MP4 file to get "moov" - "trak" - 0 - "mdia" - "minf" - "stbl". I slice a piece of the MP4 file from stbl.stco[0], of a length stbl.stsz[0] (80 and 42548 in this case). The bytes are

[0, 0, 0, 2,   9, 16, 0, 0,   166, 42, 37, 136,   128, 67, 255, 235, ...]

I still see an error message:

Failed to execute 'decode' on 'VideoDecoder': A key frame is required after configure() or flush().

What am I doing wrong? How can I find a right piece of data, that should be given to the VideoDecoder?

Ivan Kuckir
  • 2,327
  • 3
  • 27
  • 46
  • You can contact media-dev@chromium.org. It looks like this problem was also discussed in https://github.com/w3c/webcodecs/issues/220. In general, if you can provide a minimal reproduction case, this can help the team investigate more easily. – DenverCoder9 Aug 01 '22 at 09:55
  • @IvanKuckir What is the first 16 bytes of your "piece"? I wonder if track `0` has video or audio? – VC.One Aug 01 '22 at 11:35
  • @IvanKuckir Regarding your _"...and you don't know the framerate."_ . To ge the MP4's frame rate using JS then please try [my Answer here](https://stackoverflow.com/a/70895468/2057709) to see if it helps you. You want to look at: `function get_MP4_info` in there. – VC.One Aug 01 '22 at 11:59
  • Track 0 is a video track, I edited my question and added the actual bytes. – Ivan Kuckir Aug 01 '22 at 15:42

2 Answers2

2

All my steps were correct. The problem was, that the VideoDecoder missed the description parameter during configure(). Seems like no MP4 file can be decoded without it.

Use the sequence of 40 bytes at stbl.stsd.extraData.avcC as a value of the description.

Ivan Kuckir
  • 2,327
  • 3
  • 27
  • 46
  • I'm trying to do the same, but in my case `stsd` does not have `extraData` field. Do you have some working example somewhere? My goal is exactly the same as yours - capture frames and render them on canvas. – Adam Pietrasiak Dec 06 '22 at 23:07
  • I think some MP4 files can be decoded without it, depending on a specific codec that is used in a video. – Ivan Kuckir Dec 08 '22 at 07:06
0

The key frame error refers to the EncodedVideoChunk.type field, which must be "key" at the start of decoding. It's assumed that this information comes from the media container.

This is independent of the format of the bitstream; if the bitstream is Annex B then you should not set a description, and if the stream is AVC (used in MP4), you should set description.

Dan Sanders
  • 11
  • 1
  • Of course, I do sey the type to "key". – Ivan Kuckir Aug 13 '22 at 00:22
  • It's possible to get this message in Chrome when the frame fails key frame verification even if `type` is set correctly, but if the underlying reason is that it was expecting Annex B and got AVC, you should get the message "A key frame is required after configure() or flush(). If you're using AVC formatted H.264 you must fill out the description field in the VideoDecoderConfig." It looks like the specialized message for this case was added in Chrome 106.0.5221.0. – Dan Sanders Aug 15 '22 at 15:45