passing real-time metadata synchronized with video frames using webrtc and text tracks

Question

I'm using WebRTC (win c++ native client) to broadcast real-time video to peers (hosted in Chrome).
Goal: send metadata along each video frame (metadata changes at frame level).

Would it be possible to send the metadata within a text track to be consumed by a javascript at the peer side?
If not, is there an alternative way of synchronizing WebRTC real-time video with metadata?
e.g., using WebRTC DataChannel / WebSockets?

Possible duplicate of [Synchronization of data with video using WebRTC](https://stackoverflow.com/questions/30391939/synchronization-of-data-with-video-using-webrtc) — bain, Aug 28 '19 at 09:09

score 2 · Answer 1 · answered Nov 08 '19 at 04:38

2

I guess you need this features

https://webrtc.googlesource.com/src/+/77c8e65b88c9d2d95442b66ada504e0f1c553d66

Update multiplex encoder to support having augmenting data attached to the video

answered Nov 08 '19 at 04:38

M.Hefny

2,677
1
26
30

Has this ever been implemented? – Trmotta Jul 05 '22 at 19:31

score 0 · Answer 2 · answered Jul 29 '18 at 04:10

No.

At the moment, WebRTC implementation (or specification) comes with no synchronization or the ability to synchronize data with video on a frame-by-frame basis. This is something being looked at for future WebRTC specification.

There are vendors who offer such a capability in their SDKs, but this is limited to their native SDKs and doesn't work in their browser based JS SDKs.

passing real-time metadata synchronized with video frames using webrtc and text tracks

2 Answers2