4

HTML5 has <video/> element that downloads video from server, decodes it and renders. Often if not always they use hardware accelerated decoding (if available).

Is it possible to access just decoding functionality? The reason is that I'm using custom streaming protocol and so on client side I have encoded video stream that I need to decode and render.

Pure JavaScript implementations of video decoder are not applicable unfortunately as they cannot provide sufficient performance. I'm interested in HVEC or h.264 codecs only.

Andriy Tylychko
  • 15,967
  • 6
  • 64
  • 112

3 Answers3

10

Is it possible to access just decoding functionality?

Unfortunately no. We only have access to a high-level API dealing with the stream/source file agnostically with limited influence such as time-based position, play state and various events. We can draw frames to a canvas as raw RGB(A) from the current decoded frame, but that's about it.

The reason is that I'm using custom streaming protocol and so on client side I have encoded video stream that I need to decode and render

You aren't describing this protocol so we can only guess, but you might be able to build a browser compatible stream that can be used by the video element using Media Source Extensions. This allows you to build adaptive and custom streaming solutions right in the client.

Pure JavaScript implementations of video decoder are not applicable unfortunately as they cannot provide sufficient performance.

This is not necessarily true. Examples are for example pure JS implementation that decodes MPEG1 streams in real-time, both audio and video, such as this and this. Of course, this works on the very limit of what most browsers can do at present time. There is also an emscripten based H-264 decoder available that seem to also utilize the GPU via WebGL, but I cannot speak for its performance - it may be a good starting point for the next paragraph though:

A better option is to look into WebAssembly which can run pre-compiled binary code from for example C/C++ source-code. This allows you to use open-source implementations of HVEC/H.264 decoders running at native speed in the browser (careful with the licenses and terms though, especially for H.264), or use parts of software such as (linkable) ffmpeg.

I'm interested in any even non-portable solution

You may in that case want to look into building a web-extension (aka browser extensions) which can use messaging to interact with native application (the latter could be ffmpeg in this case, or a program that can deal with the stream directly).

How exactly this would work will of course depend on the protocol you're using and so forth.

Just my 2 cents based on the limited scope/description.

  • +1. As I understand the video element can be used only with HLS or MPEG-DASH both don't support low-latency streaming? I mean milliseconds of response, not seconds: https://www.wowza.com/blog/hls-latency-sucks-but-heres-how-to-fix-it. this's the only reason for a custom protocol and this discounts the video element, please correct me. unfortunately MPEG-1 is not acceptable, web-extensions are deprecated. Existing JS decoders are straightforward ports by emscripten and show bad performance on heavy streams. – Andriy Tylychko Dec 19 '17 at 11:38
  • 1
    that's a shame that functionality supported by browsers and required for a whole class of applications are not exposed, even in a non-portable way – Andriy Tylychko Dec 19 '17 at 11:40
1

After long research, decoding h264 HLS TS segment stream using hardware decoder is possible on Android browsers using media source extension (MSE). Since MSE is not supported by iOS, getting it to work on Safari in iOS seems like hit a roadblock as Apple doesn't allow access to hardware decoder via FIFO buffer or callbacks. Given, Apple's support to WebRTC, seems like only way to get to hardware decoder in iOS would be equivalent to "receiving a video call" flow except the input has to be a remote http stream and output has to go to canvas.

Karthik Vaithianathan
  • 11
  • 1
0

The solution for this problem is WebRTC. It's possible to integrate an external encoder or to use an embedded one. In browser WebRTC client uses H/W decoding. WebRTC also provides real-time streaming functionality. Compatibility is not bad.

Andriy Tylychko
  • 15,967
  • 6
  • 64
  • 112
  • Could you please extend your answer a bit? I am very interested in any ways of decoding a video on the client side (to be more specific - extracting frame sequence without drops and duplications), but I am not really sure how to apply WebRTC in this case. Do I have to setup a special server which will be serving decoded frames to the client via WebRTC? – Michael Radionov Jul 30 '18 at 15:57
  • 1
    @MichaelRadionov: WebRTC was designed for peer-to-peer video conferences, like web-browser to web-browser, and has JavaScript API. For anything less trivial its source code is open and has native C++ API (aka "Native WebRTC") plus binding for other languages. You need to stream by WebRTC and receive by embedded WebRTC client for hardware decoding in web-browser. It's real-time streaming so frame drops do occur but no duplications. I'd suggest you describe your case in detail as a separate question and I'll try to answer it. – Andriy Tylychko Jul 30 '18 at 21:57
  • I'm interested in getting the raw decoded frames directly, which would be possible with "using just decoding functionality". But from what I read it is not possible even with WebRTC. Correct me if I'm wrong, but with webRTC you are bound to use the `video` element, to get the HW-decoding done, right? – N4ppeL Jul 04 '19 at 12:55