7

For H264 encoding WebRTC uses OpenH264 which does not support hardware acceleration. There are many third party codecs included in WebRTC including WebRTC. How FFmpeg can be used instead? "is_component_ffmpeg=true" does not seem to do anything.

The goal here is to encode with hardware acceleration to have reduced latency and cpu usage. We have hardware encoder running but do not know how to plug that into webrtc. Using hardware acceleration is the closest option.

Where do we need to look at to use FFmpeg? or use externally encoded h264 data stream?

VC.One
  • 14,790
  • 4
  • 25
  • 57
SMUsamaShah
  • 7,677
  • 22
  • 88
  • 131

2 Answers2

6

We ended up modifying h264_encoder_impl by replacing all OpenH264 API calls with our own encoder calls.

WebRTC constantly keeps asking the encoder implementation to update the bitrate and framerate as it sees fit for current available bandwidth. The HW encoder we used supported updating only bitrates on the fly and that worked fine with WebRTC. Framerate was set to a fixed value.

As we did not change framerate as per wishes of WebRTC and it still worked fine, I think that encoded stream can also be sent the same way after doing only RTPFragmentation properly for given encoded buffer.

SMUsamaShah
  • 7,677
  • 22
  • 88
  • 131
  • Hi, have looks like you done outstanding job - as far as i understand WebRtc gives us only software encoding. Do you have in your plans to release it as opensource or this is proprietary software. After Hardware encoding how latency is changed in particular is it good for cloud gaming ? Thank you. – Oleksandr Papchenko Jan 09 '18 at 10:01
  • 1
    @OleksandrPapchenko The effect of doing this was immediately visible. Performance was increased and latency reduced. It was outstanding. – SMUsamaShah Jan 10 '18 at 07:50
  • thank you for response. Definitely will try this out. – Oleksandr Papchenko Jan 10 '18 at 08:11
3

We've attempted to shunt the encoding portion of the WebRTC project in the past with little luck (we wanted to pass through data that had already been encoded to multiple WebRTC clients). My impression is that it's very tightly integrated with quality of service. WebRTC wants to adjust the encoder settings based on current network traffic.

The best solution we found is to actually roll your own WebRTC using the dtlssrtpenc, nicesink, and nicesrc elements from the OpenWebRTC project:

https://github.com/EricssonResearch/openwebrtc-gst-plugins

This wasn't at all easy to do. WebRTC has a very complicated handshake and those GStreamer elements require a lot of special hookup, but it did yield the desired results.

Oh and btw our experience is that openh264 works quite well for WebRTC traffic and we ended up using it for many cases.

mpr
  • 3,250
  • 26
  • 44
  • CPU usage in our case is pretty high for real time stream, a little less than VP8 though. Does WebRTC provide some API to implement your own extension of encoder. `h264_encoder_impl` looks like a starting point for but what what should be done with it? – SMUsamaShah Jul 20 '17 at 12:28
  • There's a lot of work being done on the WebRTC code base but when I looked at it a few months back, I didn't see any practical way of shunting in a new encoder, and even if I were to do it, they weren't actively maintaining an API for it or anything like that. The WebRTC mailing list might be a good place to check if you're set on going that route. – mpr Jul 20 '17 at 17:17
  • Added Nvidia hardware encoded by modifying `h264_encoder_impl` for now. But now stuck on setting `RTPFragmentationHeader`, what does it mean? What information it expects? How it should be filled? – SMUsamaShah Aug 09 '17 at 11:57
  • From my rough reading of the RtpFragmentize() function, it looks like `SFrameBSInfo* info` drives the setup of the RTPFragmentationHeader, so if by some chance you've filled that out correctly the rest should follow. Hard to say. Have you checked the WebRTC dev group (I think it's on Google groups)? This is something the core devs can probably guide you better on. Unless it's some setting that can be tweaked in a file, StackOverflow is probably not the best place to seek help on this. – mpr Aug 09 '17 at 14:53
  • solved fragmentation header https://stackoverflow.com/questions/45632432/webrtc-what-is-rtpfragmentationheader-in-encoder-implementation – SMUsamaShah Aug 21 '17 at 12:59