How many frame need to be decoded before output callback is invoked in webcodec(avc1.640028)

Question

When I invoked

  VideoDecoder.decode(
    new EncodedVideoChunk(sample)
  );

once in a tick of requestAnimationFrame, I noticed the output callback is also invoked roughly in each requestAnimationFrame, but the video quality is poor

However, when I invoke

  VideoDecoder.decode(
    new EncodedVideoChunk(sample)
  );

multiple times in the same tick and output callback can produce the frame of good quality

Why this happen? I there anything related to the fact that decoder tend to refer to some "after" sample? It can only refer to "after" samples when the "after" sample is enqueued in the same tick. If the samples are enqueued in the next tick, the web decoder will choose to emit the frame without referring to the "late" samples?

It my guess is true, what is a good number of samples to decode in the same tick to produce a frame with good quality

The following data may provide more hint: When decode and render in different tick(decode then render), it is OK

decode:0 frame
decode:1 frame
decode:2 frame
decode:3 frame
decode:4 frame
decode:5 frame
decode:6 frame
decode:7 frame
decode:8 frame
decode:9 frame
decode:10 frame
decode:11 frame
decode:12 frame
decode:13 frame
decode:14 frame
decode:15 frame
decode:16 frame
decode:17 frame
decode:18 frame
decode:19 frame
decode:20 frame
decode:21 frame
decode:22 frame
decode:23 frame
decode:24 frame
decode:25 frame
decode:26 frame
decode:27 frame
decode:28 frame
decode:29 frame
decode:30 frame
decode:31 frame
decode:32 frame
decode:33 frame
decode:34 frame
decode:35 frame
decode:36 frame
decode:37 frame
decode:38 frame
decode:39 frame
decode:40 frame
decode:41 frame
decode:42 frame
decode:43 frame
decode:44 frame
decode:45 frame
decode:46 frame
decode:47 frame
decode:48 frame
decode:49 frame
decode:50 frame
decode:51 frame
decode:52 frame
decode:53 frame
decode:54 frame
decode:55 frame
decode:56 frame
decode:57 frame
decode:58 frame
decode:59 frame
decode:60 frame
decode:61 frame
decode:62 frame
decode:63 frame
decode:64 frame
decode:65 frame
decode:66 frame
decode:67 frame
decode:68 frame
decode:69 frame
decode:70 frame
decode:71 frame
decode:72 frame
decode:73 frame
decode:74 frame
decode:75 frame
decode:76 frame
decode:77 frame
decode:78 frame
decode:79 frame
decode:80 frame
decode:81 frame
decode:82 frame
decode:83 frame
decode:84 frame
decode:85 frame
decode:86 frame
decode:87 frame
decode:88 frame
decode:89 frame
decode:90 frame
decode:91 frame
decode:92 frame
decode:93 frame
decode:94 frame
decode:95 frame
decode:96 frame
decode:97 frame
decode:98 frame
decode:99 frame
render:0 frame
render:3 frame
render:2 frame
render:4 frame
render:1 frame
render:7 frame
render:6 frame
render:8 frame
render:5 frame
render:11 frame
render:10 frame
render:12 frame
render:9 frame
render:15 frame
render:14 frame
render:16 frame
render:13 frame
render:19 frame
render:18 frame
render:20 frame
render:17 frame
render:23 frame
render:22 frame
render:24 frame
render:21 frame
render:27 frame
render:26 frame
render:28 frame
render:25 frame
render:31 frame
render:30 frame
render:32 frame
render:29 frame
render:35 frame
render:34 frame
render:36 frame
render:33 frame
render:39 frame
render:38 frame
render:40 frame
render:37 frame
render:43 frame
render:42 frame
render:44 frame
render:41 frame
render:47 frame
render:46 frame
render:48 frame
render:45 frame
render:51 frame
render:50 frame
render:52 frame
render:49 frame
render:55 frame
render:54 frame
render:56 frame
render:53 frame
render:59 frame
render:58 frame
render:60 frame
render:57 frame
render:63 frame
render:62 frame
render:64 frame
render:61 frame
render:67 frame
render:66 frame
render:68 frame
render:65 frame
render:71 frame
render:70 frame
render:72 frame
render:69 frame
render:75 frame
render:74 frame
render:76 frame
render:73 frame
render:79 frame
render:78 frame
render:80 frame
render:77 frame
render:83 frame
render:82 frame
render:84 frame
render:81 frame
render:87 frame
render:86 frame
render:88 frame
render:85 frame
render:91 frame
render:90 frame
render:92 frame
render:89 frame
render:95 frame
render:94 frame
render:96 frame

However when decode and render at the same tick(decode and render at the same time) the frame quality is poor

decode:0 frame
decode:1 frame
decode:2 frame
decode:3 frame
decode:4 frame
render:0 frame
decode:5 frame
render:3 frame
decode:6 frame
render:2 frame
decode:7 frame
render:4 frame
decode:8 frame
render:1 frame
decode:9 frame
render:7 frame
decode:10 frame
render:6 frame
decode:11 frame
render:8 frame
decode:12 frame
render:5 frame
decode:13 frame
render:11 frame
decode:14 frame
render:10 frame
decode:15 frame
render:12 frame
decode:16 frame
render:9 frame
decode:17 frame
decode:18 frame
render:15 frame
render:14 frame
decode:19 frame
render:16 frame
decode:21 frame
render:13 frame
decode:22 frame
render:19 frame
decode:23 frame
render:18 frame
decode:24 frame
render:17 frame
decode:25 frame
render:23 frame
decode:26 frame
render:22 frame
decode:27 frame
render:24 frame
decode:28 frame
render:21 frame
decode:29 frame
render:27 frame
decode:30 frame
render:26 frame
decode:31 frame
render:28 frame
decode:32 frame
decode:33 frame
render:25 frame
render:31 frame
decode:34 frame
render:30 frame
decode:35 frame
render:32 frame
decode:36 frame
render:29 frame
decode:37 frame
render:35 frame
decode:38 frame
render:34 frame
decode:39 frame
render:36 frame
decode:40 frame
render:33 frame
decode:41 frame
render:39 frame
decode:43 frame
render:38 frame
decode:44 frame
render:40 frame
decode:45 frame
render:37 frame
decode:46 frame
render:43 frame
decode:47 frame
render:44 frame
decode:48 frame
render:41 frame
decode:49 frame
render:47 frame
decode:50 frame
render:46 frame
decode:51 frame
decode:52 frame
render:48 frame
decode:53 frame
render:45 frame
decode:54 frame
render:51 frame
decode:55 frame
render:50 frame
decode:56 frame
render:52 frame
render:49 frame
decode:57 frame
render:55 frame
decode:58 frame
render:54 frame
decode:59 frame
render:56 frame
decode:60 frame
render:53 frame
decode:61 frame
render:59 frame
decode:62 frame
render:58 frame
decode:63 frame
render:60 frame
decode:64 frame
render:57 frame
decode:65 frame
render:63 frame
decode:66 frame
render:62 frame
decode:67 frame
render:64 frame
decode:68 frame
render:61 frame
decode:69 frame
render:67 frame
decode:70 frame
render:66 frame
decode:71 frame
render:68 frame
decode:72 frame
render:65 frame
decode:73 frame
render:71 frame
decode:74 frame
render:70 frame
decode:75 frame
render:72 frame
decode:76 frame
render:69 frame
decode:77 frame
render:75 frame
decode:78 frame
render:74 frame
decode:79 frame
decode:80 frame
render:76 frame
render:73 frame
decode:81 frame
render:79 frame
decode:82 frame
render:78 frame
decode:83 frame
render:80 frame
decode:84 frame
render:77 frame
decode:85 frame
render:83 frame

Answer 1

To better pin-point your problem, I would start by making sure that requestAnimation Frame is in sync with the decoder. Strangely, the output function is not the right place to do it, since the frame building is asynchronous. For that very reason, the videoDecoder has an event called "ondequeue" which will fire only when the frame is ready.

So, my suggestion for you is controlling your animation tick with de videoDecoder.ondequeue event (so that the decoder and the requestAnimationFrame works in sync). It is also good practice to allow the decoder to buffer some frames before start rendering, like that:

videoDecoder.ondequeue = () => {
    decodeFrames++
    if(decodedFrames>??){
        requestAnimationFrame();
        decodedFrames--
    }
};