H264 WebRTC video streamed from ffmpeg through Janus is very choppy on playback

Question

Trying to stream video through following chain: h264/mp4 file on local instance storage (AWS)->ffmpeg->rtp->Janus on same instance->WebRTC playback (Chrome/mac). Resulting video is choppy even as none of the resources seem overloaded (CPU/memory/network bandwidth on any of the systems involved). I also use a Coturn TURN server, it is also not loaded at all and bandwidth is plentiful.

Tried switching codecs and it didn't help apart from vp8 which while worked (kind of - choppiness was still there but very rare and acceptable), resulted in such a high CPU consumption that practically it's unacceptable.

ffmpeg -re -stream_loop -1 -i ./short.mp4 -s 426x240 -c:v libx264 -profile:v baseline -b:v 1M -r 24 -g 60 -an -f rtp rtp://127.0.0.1:5004

resulting SDP is:

o=- 0 0 IN IP4 127.0.0.1
s=No Name
c=IN IP4 127.0.0.1
t=0 0
a=tool:libavformat 58.20.100
m=video 5004 RTP/AVP 96
b=AS:1000
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1

stream is set up with Janus API as

            "janus" : "message",
            "transaction" : 'Transaction',
        "body": {
                "request" : "create",
                "type" : "rtp",
                "id" : newId,
                "name": streamId+newId,
                "audio": false,
                "video": true,
                "description" : streamId+newId,
                "videoport" : 5000+newId*4,
                "videopt" : 96,
                "videortpmap": "H264/90000",
                "secret" : "adminpwd"
            }
        }

Tried various options of bw, doesn't help at all. Changing -g (GOP size) to lower values can make choppiness shorter in duration but more frequent. At -g 3 or 4 it is acceptable but the bitrate for tolerable quality, predictably, becomes insane.

Expected result: video plays without choppiness.

My theory of it is it could be one of the following:

• Either ffmpeg provides data in a way buffer is too small so sometimes Janus needs to send a next packet while it's not ready yet, starving buffer and resulting in interruption - so maybe there is a way to make ffmpeg encode into some kind of a short (half-second or so? buffer to regulate flow). How?

• Or H264 works too poorly over UDP as it is and there is nothing i could do. Then i got to switch to TCP, but so far attempts to do this has been unsuccessful.

Answer 1

The solution proved to be beautiful in it's obviousness. ffmpeg sent stream to Janus as RTP, Janus sent it further to viewers, obviously, as SRTP, because this is WebRTC and it is always encrypted. Which added a bunch of bytes to each packet as encryption overhead. In some cases, it meant packets going over the MTU and discarded - each time it happened, there was a visible jerk in video.

Simple addition of ?pkt_size=1300 to output RTP URL of ffmpeg removed the problem.

Thanks to Lorenzo Miniero of Meetecho for figuring this out.

Answer 2

ffmpeg is optimized for outputting frames in chunks, not for outputting individual coded frames. The muxer, in your case the rtp muxer, normally buffers data before flushing to output. So ffmpeg is not optimized for real-time streaming that requires more or less frame-by-frame output. WebRTC, however, really needs frames arriving in real-time, so if frames are sent in bunches, it may discard the "late" frames, hence the choppiness.

However, there is an option in ffmpeg, to set muxer's buffer size to 0, that works nice. It is:

-max_delay 0

Also, for WebRTC, you want to disable b-frames and append SPS-PPS to every key frame:

-bf 0 +global_header -bsf:v "dump_extra=freq=keyframe"