How to reduce Android hw video decoding latency? 30ms for H264 720p?

Question

I am trying to test if it is possible to use HW docoding for realtime communication, say 100ms for 720p client to client delay, 40ms for system delay and 60ms for network, iOS with VT is already OK. When I tried it with AMediaCodec, it looks like the codec always buffers more than 8 frames before the first output is available. Keep calling AMediaCodec_queueInputBuffer to input video frames AMediaCodec_dequeueOutputBuffer to handle the output. But I will not get any avaliable outputs until 8 frames or more are feeded in. Got AMEDIACODEC_INFO_TRY_AGAIN_LATER instead. Tried both H264 high profile and baseline. I got a DPB(Decoded Picture Buffer) Size 8 when I check the dumped h264 stream with Codecvisa, looks like a hint for the extra buffering?

I also performed some simple testing of MoviePlayer.java in the Grafika MediaCodec sample code. I am foucing on how many output buffers the codec holds after the initial decoding lag, the initial delay is what I can accept as it is only on the beginning. I add an extra_buffer_cnt which ++ when queueInputBuffer succuss and -- when dequeueOutputBuffer, and I also add a while loop to call dequeueOutputBuffer to make sure the dequeueOutputBuffer can be called in time. From the logs, I can see extra buffer is always 8 for some strems and can be 0 for the others.

12-07 10:57:45.614   : decoderInputBuffers pts 0
    submitted frame 0 to dec, size=40658
12-07 10:57:45.634   : no output from decoder available
    loop
    decoderInputBuffers pts 98000
    submitted frame 1 to dec, size=17438
12-07 10:57:45.664   : no output from decoder available
    loop
    decoderInputBuffers pts 200000
    submitted frame 2 to dec, size=17410
    decoder output buffers changed
12-07 10:57:45.674   : no output from decoder available
    loop
    decoderInputBuffers pts 351000
    submitted frame 3 to dec, size=22043
12-07 10:57:45.704   : no output from decoder available
    loop
    decoderInputBuffers pts 450000
    submitted frame 4 to dec, size=21515
12-07 10:57:45.724   : no output from decoder available
    loop
    decoderInputBuffers pts 550000
    submitted frame 5 to dec, size=21449
12-07 10:57:45.754   : no output from decoder available
    loop
    decoderInputBuffers pts 650000
    submitted frame 6 to dec, size=21640
12-07 10:57:45.774   : no output from decoder available
    loop
    decoderInputBuffers pts 750000
    submitted frame 7 to dec, size=25636
12-07 10:57:45.804   : no output from decoder available
    loop
    decoderInputBuffers pts 848000
    submitted frame 8 to dec, size=21466
12-07 10:57:45.814   : no output from decoder available
    decoder output format changed: {mime=video/raw, crop-top=0, crop-right=359, slice-height=640, color-format=2141391876, height=640, width=368, what=1869968451, buffer-size=380928, crop-bottom=639, crop-left=0, stride=384}
    loop
    decoderInputBuffers pts 950000
12-07 10:57:45.824   : submitted frame 9 to dec, size=25224
    startup lag 210.543229 ms
    surface decoder given buffer 14 (size=8)
    dequeueOutputBuffer pts 0 extra buffer 9
12-07 10:57:45.834   : surface decoder given buffer 13 (size=8)
    dequeueOutputBuffer pts 98000 extra buffer 8
    loop
    decoderInputBuffers pts 1046000
    submitted frame 10 to dec, size=25172
12-07 10:57:45.844   : surface decoder given buffer 12 (size=8)
    dequeueOutputBuffer pts 200000 extra buffer 8
12-07 10:57:45.864   : no output from decoder available
    loop
    decoderInputBuffers pts 1149000
    submitted frame 11 to dec, size=25137
12-07 10:57:45.874   : surface decoder given buffer 11 (size=8)
    dequeueOutputBuffer pts 351000 extra buffer 8
12-07 10:57:45.884   : no output from decoder available
    loop
    decoderInputBuffers pts 1248000
    submitted frame 12 to dec, size=25343
12-07 10:57:45.894   : surface decoder given buffer 10 (size=8)
    dequeueOutputBuffer pts 450000 extra buffer 8
12-07 10:57:45.914   : no output from decoder available
    loop
    decoderInputBuffers pts 1400000
    submitted frame 13 to dec, size=22197
12-07 10:57:45.924   : surface decoder given buffer 9 (size=8)
    dequeueOutputBuffer pts 550000 extra buffer 8
12-07 10:57:45.934   : no output from decoder available
    loop
    decoderInputBuffers pts 1499000
    submitted frame 14 to dec, size=26267
12-07 10:57:45.944   : surface decoder given buffer 8 (size=8)
    dequeueOutputBuffer pts 650000 extra buffer 8
12-07 10:57:45.954   : no output from decoder available
    loop
    decoderInputBuffers pts 1600000
    submitted frame 15 to dec, size=25947
12-07 10:57:45.964   : surface decoder given buffer 7 (size=8)
    dequeueOutputBuffer pts 750000 extra buffer 8
12-07 10:57:45.984   : loop
    decoderInputBuffers pts 1700000
    submitted frame 16 to dec, size=26060
12-07 10:57:45.994   : surface decoder given buffer 6 (size=8)
    dequeueOutputBuffer pts 848000 extra buffer 8
12-07 10:57:46.004   : no output from decoder available
    loop
    decoderInputBuffers pts 1799000
    submitted frame 17 to dec, size=26003
12-07 10:57:46.014   : surface decoder given buffer 5 (size=8)
    dequeueOutputBuffer pts 950000 extra buffer 8
12-07 10:57:46.024   : no output from decoder available
    loop
    decoderInputBuffers pts 1900000
    submitted frame 18 to dec, size=22085
12-07 10:57:46.034   : surface decoder given buffer 13 (size=8)
    dequeueOutputBuffer pts 1046000 extra buffer 8
12-07 10:57:46.044   : no output from decoder available
    loop
    decoderInputBuffers pts 2012000
12-07 10:57:46.054   : submitted frame 19 to dec, size=26195
    surface decoder given buffer 12 (size=8)
    dequeueOutputBuffer pts 1149000 extra buffer 8
12-07 10:57:46.074   : no output from decoder available
    loop
    decoderInputBuffers pts 2097000
    submitted frame 20 to dec, size=25963
12-07 10:57:46.084   : surface decoder given buffer 11 (size=8)
    dequeueOutputBuffer pts 1248000 extra buffer 8
12-07 10:57:46.094   : no output from decoder available
    loop
    decoderInputBuffers pts 2200000
    submitted frame 21 to dec, size=25869

Please see *[How to reduce latency in MediaCodec video/avc decoding](https://stackoverflow.com/a/21444370/192373)*. Spoiler: the HW may need more time to decode the first frame. — Alex Cohn, Dec 04 '18 at 09:02
The input steam is zero-latency, I checked it with an iOS device, less than 100ms for end to end. — Wanpeng, Dec 04 '18 at 09:43
The extra buffering was not only for first frame but all of them — Wanpeng, Dec 04 '18 at 09:43
Sorry, I tested it again and looks it was only about the first frame — Wanpeng, Dec 05 '18 at 13:04
Thanks for update. If @fadden's solution helps, your question can be closed, can't it? — Alex Cohn, Dec 05 '18 at 15:29
@fadden's solution did help, but when testing different video streams I can get different result, I think this question will not be closed until I finger out what's going on for different streams. — Wanpeng, Dec 07 '18 at 01:59
Changed max_dec_frame_buffering to 1 in SPS, I was able to get zero-latency on VIVO phone. But it still does not work on a Huawei phone. — Wanpeng, Dec 11 '18 at 02:30
Maybe this latency was the real cause of the recent arrest of Huawei exec ;) — Alex Cohn, Dec 11 '18 at 07:16

score 0 · Answer 1 · answered Jan 18 '23 at 04:40

You need to configure customized(or KEY_LOW_LATENCY if it is supported) low latency parameters for different cpu venders. It is a common problem for android phone.

For huawei phone, it is

videoFormat.setInteger("vendor.hisi-ext-low-latency-video-dec.video-scene-for-low-latency-req", 1);
videoFormat.setInteger("vendor.hisi-ext-low-latency-video-dec.video-scene-for-low-latency-rdy", -1);

Check this code for details. https://github.com/moonlight-stream/moonlight-android/blob/master/app/src/main/java/com/limelight/binding/video/MediaCodecHelper.java

How to reduce Android hw video decoding latency? 30ms for H264 720p?

1 Answers1