Synchronizing Audio and Video Playback when playing frame by frame / by sample

Question

I am currently in the process of writing an open source AviSynth editor with C#/WPF with video AND audio preview. I already got all the AviSynth stuff working and I am able to query AviSynth to get a frame (by framenumber, as bitmap) and to get audio samples (by startindex and number of samples).

Now I am struggling with the playback in my application. The first thing I would like to get working is to play a video in real-time. The problem is in video/audio synchronization. When I just play one frame and the according audio one by one, there is a noticable "gap" between the audio segments and the only way around this I found is by using a buffered playback like NAudios BufferedWaveProvider.

So there I can buffer multiple segments (so samples for multiple frames) and they play nicely. What I am now missing is the synchronization of the audio segments with the video playback.

My current approach is to have 3 threads:

• 1 thread that fills a queue with audio samples and frames (for about 1 second ahead)
• 1 thread that just plays the audio that is in the BufferedWaveProvider
• 1 thread that is responsible to get objects out of the queue and display them or add them to the BufferedWaveProvider

So thread 1 and 2 seems fine. The queue always contains elements and all samples added to the BufferedWaveProvider are played fine. The problem is in thread 3: I somehow need to be able to synchronize the video and audio there. I tried various sleeps (like until the audiobuffer which is playing is almost empty or to wait according to the framerate) but it is always going out of sync or too slow so the audio contains gaps.

I am missing the base concept on how to do audio/video synchronization in such a szenario and could not find much online and would really appreciate some help here.

For completeness, here are my 3 threads that I currently use:

private async Task FetchDataLoop()
{
    while (true)
    {
        var frameNumberToBuffer = buffer.CurrentLastBufferedFrame + 1;
        var img = clip.ReadFrame(frameNumberToBuffer);
        var snd = clip.ReadAudio(frameNumberToBuffer * clip.SamplesPerFrame, clip.SamplesPerFrame);
        var newObject = new BufferObject
        {
            VideoData = img,
            AudioData = snd
        };
        buffer.BufferObjects.Add(newObject);
        buffer.CurrentLastBufferedFrame = frameNumberToBuffer;
    }
}

private async Task SoundPlayerLoop()
{
    while (true)
    {
        using (var outputDevice = new WaveOutEvent())
        {
            outputDevice.Init(bufferedWaveProvider);
            outputDevice.Play();
            while (outputDevice.PlaybackState == PlaybackState.Playing)
            {
                // As it is an endless stream, this is never hit
                await Task.Delay(100);
            }
        }
    }
}

private async Task PlayLoop()
{
    var lastFrame = DateTime.Now;
    while (true)
    {
        var bufferedObject = buffer.BufferObjects.Take();

        bufferedWaveProvider.AddSamples(bufferedObject.AudioData, 0, bufferedObject.AudioData.Length);
        Application.Current.Dispatcher.Invoke(() =>
        {
            vm.Image = BitmapToImageSource(bufferedObject.VideoData);
        });

        // TODO: Somehow synchronize the audio/video here
    }
}