How to eliminate artifacting with fast update WriteableBitmap

Question

I'm trying to implement a video renderer with a WriteableBitmap.

My current implementation works fine for full screen 60fps content, but if I scrub the video (drag the playback back/forth to a new time) the renderer starts updating the WriteableBitmap after the second render cycle started, causing it to appear with artifacting (the frame rendering engine uses parallelism).

I tried implementing a double buffer to see if it could help, and it did to a certain point, but the artifacting continues.

public WriteableBitmap RenderedImage
{
    get => _frontBuffer;
    set => SetProperty(ref _frontBuffer, value);
}

public long CurrentRenderTime
{
    get => _currentRenderTime;
    set
    {
        if (SetProperty(ref _currentRenderTime, value))
            if (RenderedImage != null)
                _event.Set();
    }
}

private void RenderFrame()
{
    while (!_renderCancellationSource.IsCancellationRequested)
    {
        //Wait until the CurrentTime changes to render again.
        _event.WaitOne();

        //TEST: No noticeable difference with this lock.
        lock (Lock)
        {
            var currentTimestamp = CurrentRenderTime;

            //Draw the background and all tracks.
            DrawBackground(_backBufferAddress, _backStride, _backPixelWidth, _backPixelHeight);

            foreach (var track in Tracks)
                track.RenderAt(_backBufferAddress, _backStride, CurrentRenderTime);

            //Swap the back buffer with the front buffer.
            (_frontBuffer, _backBuffer) = (_backBuffer, _frontBuffer);
            (_frontBufferAddress, _backBufferAddress) = (_backBufferAddress, _frontBufferAddress);
            (_frontStride, _backStride) = (_backStride, _frontStride);
            (_frontPixelWidth, _backPixelWidth) = (_backPixelWidth, _frontPixelWidth);
            (_frontPixelHeight, _backPixelHeight) = (_backPixelHeight, _frontPixelHeight);

            //TEST: No difference with this check.
            if (currentTimestamp != CurrentRenderTime)
                continue;

            Application.Current.Dispatcher.Invoke(() =>
            {
                RenderedImage.Lock(); 
                RenderedImage.AddDirtyRect(new Int32Rect(0, 0, RenderedImage.PixelWidth, RenderedImage.PixelHeight));
                RenderedImage.Unlock();

                OnPropertyChanged(nameof(RenderedImage));

                _eventRenderer.Set();
            }, DispatcherPriority.Render);

            //TEST: No difference with this wait.
            _eventRenderer.WaitOne();
        }
    }
}

Lines marked with TEST are last-minute changes that provided no noticeable results.

More details about the implementation

The rendering must occur on another thread, and it's triggered by changes in the CurrentRenderTime by using a AutoResetEvent.

Pseudo-code:

Calculate timings, variable or fixed framerate.
Iterate through the timings.
Call render.
Wait (between-frame timing - time it took to render).

The WriteableBitmap is being displayed by a WPF Image element.

The playback mechanism is not the issue here, and anyway I use a fine-tuned timer resolution (PInvoking timeBeginPeriod).

This is the minimal example:

//This method runs in a separated thread.
private void RenderFrame()
{
    while (!_renderCancellationSource.IsCancellationRequested)
    {
        //Wait until the CurrentTime changes to render again.
        _event.WaitOne();

        lock (Lock)
        {
            //Draw the background and all tracks.
            DrawBackground(_backBufferAddress, _backStride, _backPixelWidth, _backPixelHeight);

            foreach (var track in Tracks)
                track.RenderAt(_backBufferAddress, _backStride, CurrentRenderTime);

            //Swap the back buffer with the front buffer.
            (_frontBuffer, _backBuffer) = (_backBuffer, _frontBuffer);
            (_frontBufferAddress, _backBufferAddress) = (_backBufferAddress, _frontBufferAddress);
            (_frontStride, _backStride) = (_backStride, _frontStride);
            (_frontPixelWidth, _backPixelWidth) = (_backPixelWidth, _frontPixelWidth);
            (_frontPixelHeight, _backPixelHeight) = (_backPixelHeight, _frontPixelHeight);

            //WriteableBitmap Lock/Unlock needs to run on the same thread as the UI.
            Application.Current.Dispatcher.Invoke(() =>
            {
                RenderedImage.Lock(); 
                RenderedImage.AddDirtyRect(new Int32Rect(0, 0, RenderedImage.PixelWidth, RenderedImage.PixelHeight));
                RenderedImage.Unlock();

                //Since I swapped the buffers, I need to tell WPF about that.
                OnPropertyChanged(nameof(RenderedImage));
            }, DispatcherPriority.Render);
        }
    }
}

Possible solution

One solution, but less performant would be to copy data between buffers, instead of switching them.

private void RenderFrame()
{
    while (!_renderCancellationSource.IsCancellationRequested)
    {
        _event.WaitOne();

        lock (Lock)
        {
            DrawBackground(_backBufferAddress, _backStride, _backPixelWidth, _backPixelHeight);

            foreach (var track in Tracks)
                track.RenderAt(_backBufferAddress, _backStride, Project.Width, Project.Height, CurrentRenderTime);

            Kernel32.CopyMemory(_frontBufferAddress, _backBufferAddress, (uint)(_backPixelHeight * _backStride));

            Application.Current.Dispatcher.Invoke(() =>
            {
                RenderedImage.Lock();
                RenderedImage.AddDirtyRect(new Int32Rect(0, 0, RenderedImage.PixelWidth, RenderedImage.PixelHeight));
                RenderedImage.Unlock();
            }, DispatcherPriority.Render);
        }
    }
}

score 0 · Answer 1 · answered Apr 21 '23 at 14:39

0

You should most likely follow the workflow from the documentation:

For greater control over updates, and for multi-threaded access to the back buffer, use the following workflow.

Call the Lock method to reserve the back buffer for updates.

Obtain a pointer to the back buffer by accessing the BackBuffer property.

Write changes to the back buffer. Other threads may write changes to the back buffer when the WriteableBitmap is locked.

Call the AddDirtyRect method to indicate areas that have changed.

Call the Unlock method to release the back buffer and allow presentation to the screen.

keeping _backBufferAddress as a field is most likely incorrect, since it may change unless you have locked the bitmap. WriteableBitmap is already using double buffering internally, so your manual attempt at doing this is likely doing more harm than anything.

so your code should look something like

myWriteableBitmap.Lock();
try
{
   var backBufferPtr = (byte*)myWriteableBitmap.BackBuffer;
 
   DrawBackground(backBufferPtr , myWriteableBitmap.BackBufferStride, myWriteableBitmap.PixelWidth, myWriteableBitmap.PixelHeight);

    foreach (var track in Tracks)
        track.RenderAt(backBufferPtr , myWriteableBitmap.BackBufferStride, CurrentRenderTime);

    myWriteableBitmap.AddDirtyRect(new Int32Rect(0, 0, myWriteableBitmap.PixelWidth, myWriteableBitmap.PixelHeight));
}
finally
{
   myWriteableBitmap.Unlock();
}

Since you do not show anything how rendering is initiated it is difficult to tell if it is correct. Adding locks/checks/resetevents randomly are most likely not helpful, you need to understand the actual problem if you want to solve it. . If you want exact timings I would expect multi media timers to be the most correct solution, but as far as I know there are no managed API, so you may need to write or find a wrapper around the native API.

answered Apr 21 '23 at 14:39

JonasH

28,608
2
10
23

"there are no managed API" - `Stopwatch`. – Blindy Apr 21 '23 at 14:52
To be clear, up to that point your answer is correct. I'd throw in a few more red flags in his code, like invoking across threads and that random mutex protecting the loop against... itself? As if it was running in *two* threads rendering on the third Gui one? Such an incredibly weird design. – Blindy Apr 21 '23 at 14:54
@Blindy, a stopwatch provides *timing*, but is not a *timer*. I.e. it measures time with very high precision, but cannot raise events at a precise interval. To use a stopwatch as a timer you would need to do a spinloop, wasting cpu. All other timers, including Thread.Sleep, has a resolution from 1ms to 16ms depending on the system. The media timers is the only API I know of that is guaranteed to provide high resolution events. – JonasH Apr 21 '23 at 15:04
Oh wait you were talking about timer events? I thought you were talking about `timeGetTime()` which is what `Stopwatch` wraps internally. OP don't listen to that part of the advice, you cannot write a real-time high-frequency application with timer events in Windows (or any protected OS for that matter). You have to use a mix of thread yielding (`Sleep`, up to the sleep resolution) and spin-waiting (`while` loop the rest of the way) instead. – Blindy Apr 21 '23 at 15:09
@Blindy, or maybe use the API intended for applications that require high high resolution timing events? See [High resolution timer in c#](https://stackoverflow.com/questions/24839105/high-resolution-timer-in-c-sharp). That should be sufficient for *media playback* (I guess thats why it is named the way it is). – JonasH Apr 21 '23 at 15:14
@Blindy @JonasH Code with `TEST` above them are test stuff that I added to check if it would make a difference. My playback implementation is fine, the issue is if I change time manually. – Nicke Manarin Apr 21 '23 at 16:58
The main render loop is composed of: Loop, Wait for a render request, Write background, Render tracks, swap buffer, tell UI that bitmap was updated. – Nicke Manarin Apr 21 '23 at 16:59
Also, the `WriteableBitmap` back buffer address doesn't change. I'm able to use it throughout the lifecycle of the window with no problem. – Nicke Manarin Apr 21 '23 at 17:14
@NickeManarin that still sounds very odd, WPF already have a render thread, that you have no control over. Are you just using a writeable bitmap as a intermediate buffer? If so you should probably just use a regular array instead to avoid confusing readers. – JonasH Apr 24 '23 at 06:59
@NickeManarin if the idea is to write something like a game where you maintain full control over rendering, you might want to use a lower level api that give you direct control over the frame buffer, probably using some wrapper over directX/OpenGL. If you are using WPF as it is intended to be used you rarely render anything directly, just update UI objects that are then rendered by wpf. – JonasH Apr 24 '23 at 09:20
@JonasH It's a custom video renderer and it's rendered in another thread to prevent UI from hanging. The `WriteableBitmap` is the canvas. – Nicke Manarin Apr 25 '23 at 13:28
@NickeManarin That still does not make any sense to me. You can update a `WriteableBitmap` from a background thread. And that does all the backbuffering and other stuff for you. I would guess it swap the internal buffers when calling unlock, but I have not checked the implementation. In any case, there is no need to involve the UI thread. I would highly recommend reading the documentation for writeableBitmap. – JonasH Apr 25 '23 at 13:49

How to eliminate artifacting with fast update WriteableBitmap

More details about the implementation

Possible solution

1 Answers1