3

It seems that Javascript does not provide any way to reliably seek a precise frame. I'm therefore thinking that the last solution may be to use WebCodec. However, it's not clear to me how to use WebCodec to play a video or seek a precise frame. Would it be possible to have an example where I can play a video and go to a precise frame (if you can show how to sync audio it would be awesome, but I don't need it right now)?

tobiasBora
  • 1,542
  • 14
  • 23
  • Does this answer your question? [Javascript: frame precise video stop](https://stackoverflow.com/questions/70613008/javascript-frame-precise-video-stop) – gbjbaanb Mar 25 '22 at 13:05
  • @gbjbaanb No: I am the creator of this other question, and this question shows how to stop precisely a video, but not how to seek precisely. It may be possible to seek a bit ahead of time, play the video, and stop precisely, but it's quite a dirty solution. – tobiasBora Mar 25 '22 at 13:08
  • It's possible with WebCodecs, but keep in mind that WebCodecs is only one component (and a very low level one) to build this on your own. You also need to parse the container in case of a file, which is not something you can do with WebCodecs alone. Perhaps this answer explains enough details to give you a solid understanding of the topic: https://stackoverflow.com/a/9958424/2788872 – John Weisz Jun 29 '22 at 11:31
  • @JohnWeisz Thanks for the interesting reference. Unfortunately, this mostly describe "how" things are encoded, but does not explain the tools offer by javascript to deal with them. Like how to demux a file, how to deal with interleaved video/audio, how to decode, how to play at the appropriate FPS and sync with audio (after the decoding, should I "just" display the frames one after the others using a timeout or is there a more precise method to apply? (I tried in python, FPS were not accurate) how should I handle audio?) I guess part of this question is about understanding the above concepts. – tobiasBora Jun 29 '22 at 17:40
  • Note to myself: maybe give a look/try to https://github.com/phoboslab/jsmpeg – tobiasBora Jun 30 '22 at 19:54
  • Another ref https://www.w3.org/2021/03/media-production-workshop/talks/chris-cunningham-hello-webcodecs.html – tobiasBora Jul 02 '22 at 10:33
  • There's also mp4box.js which can be used to demux, but the API seems a bit confusing. I'm wondering what did you end up doing since I'm facing the same problem now – Anton Mihaylov Nov 29 '22 at 19:52
  • @AntonMihaylov yes, I saw that. So I finally end-up writting my code in python as web interface was not mandatory. But I still plan to dig into it at some point. My best bet would be to try WebCodecs + some libraries for the demux (like mp4box.js as you pointed out, or maybe the javascript ffmpeg version… not exactly sure yet what's the best strategy) – tobiasBora Nov 30 '22 at 18:55
  • I see, I played around more with making it work on the browser and mp4box.js did work well. This example was of great help if you need it https://github.com/w3c/webcodecs/tree/main/samples/video-decode-display The downside is of course that it supports only mp4. Maybe using ffmpeg wasm (or js) just for demuxing and WebCodecs for decoding is an option. – Anton Mihaylov Dec 01 '22 at 00:38
  • @AntonMihaylov Nice! Do you have the code somewhere? In particular, I'm also curious to know how you synchronize to ensure 24fps (with python my fps was not really stable from one day to another). Also, if you know how to synchronize with sound that would be really cool. – tobiasBora Dec 01 '22 at 03:07
  • 1
    Here's a gist. https://gist.github.com/antonmihaylov/a708b7e93e71513a048d6615098198ea I don't synchronize, I just have a pipeline that requests new frames constantly by calling getFrameData(..., {frame: currentFrame }). So the pipeline can be set to run at however fps you need and the decoder will do its best to get the closest frame at that time. About sound - I have no idea, but it will soon be on my to-do list – Anton Mihaylov Dec 01 '22 at 11:17
  • Actually, I do have another pipeline for real-time previews that does synchronize the fps. It uses a loop that runs every X ms, imitating a fixed frames per seconds. On every tick it calculates which frame should be rendered and calls getFrameData(..., {frame: x}). It uses the Ticker class from Pixi.Js. But I imagine it's not that complicated to do a custom one, I just happened to be using pixi for other things as well – Anton Mihaylov Dec 01 '22 at 11:30
  • Thanks! Let me know if you solve the sound issue – tobiasBora Dec 01 '22 at 11:59

0 Answers0