I want to make a video player that displays more video streams at the same time using the mkv format.
I build my app so that when it decodes an Mkv video it puts all frames from a cluster in memory, from what I observe when I read a video with only one video stream, every cluster except the last one has equal numbers of frames.
I want to merge tow mkv files in one, but at the same time, I want every cluster from the video to have equal numbers of frames from both tracks.
Is this thing possible using FFmpeg or another tool?