Show byte array image in WPF with high refresh rate

Question

Context

I have a Basler camera that throw an event when a new image is captured.
In the event arg, I can get the image grabbed as a byte array.

I have to do computation on this image and then show it in a WPF application. The camera refresh rate is up to 40FPS.

Issue and found solution

A solution to convert a byte array to a WPF image can be found here : Convert byte array to image in wpf
This solution is great to convert only one time the byte array, however I feel like there is a lot of memory loss to do it at 40FPS. A new BitmapImage() is created every time and can't be disposed.

Would there be a better solution to display in WPF a byte array that changes up to 40 FPS ? (the way the problem is handled can be completely rethought)

Code

This solution to show the camera stream in WPF works, but the BitmapImage image = new BitmapImage(); line doesn't look good to me.

private void OnImageGrabbed(object sender, ImageGrabbedEventArgs e)
{
    // Get the result
    IGrabResult grabResult = e.GrabResult;
    if (!grabResult.GrabSucceeded)
    {
        throw new Exception($"Grab error: {grabResult.ErrorCode} {grabResult.ErrorDescription}");
    }

    // Make process on the image
    imageProcessor.Process(grabResult);

    // Convert grabResult in BGR 8bit format
    using Bitmap bitmap = new Bitmap(grabResult.Width, grabResult.Height, PixelFormat.Format32bppRgb);
    BitmapData bmpData = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.ReadWrite, bitmap.PixelFormat);
    IntPtr ptrBmp = bmpData.Scan0;
    converter.Convert(ptrBmp, bmpData.Stride * bitmap.Height, grabResult);
    bitmap.UnlockBits(bmpData);

    // Creat the BitmapImage
    BitmapImage image = new BitmapImage(); // <-- never Disposed !
    using (MemoryStream memory = new MemoryStream())
    {
        bitmap.Save(memory, ImageFormat.Bmp);
        memory.Position = 0;
        image.BeginInit();
        image.StreamSource = memory;
        image.CacheOption = BitmapCacheOption.OnLoad;
        image.EndInit();
        image.Freeze();
    }
    LastFrame = image; // View is binded to LastFrame
}

Answer 1

I would suggest that you use a WriteableBitmap to display the result. This avoids the need to reallocate the UI image. If the pixel format in your source matches the one in the bitmap you can simply use WritePixels to update the image.

Note that you can only modify WriteableBitmap from the main thread, and the ImageGrabbed event will be raised on a background thread. And the grabResult will be disposed of once the event handler returns. So you will need to ask the UI thread to do the actual updating, and you will need a intermediate buffer for this. But this buffer can be pooled if needed.

An alternative might be to write your own loop, calling RetrieveResult repeatedly, this would let you dispose the grab results manually, after the UI has been updated. It might also be possible to keep a pool of WriteableBitmaps, I guess it should be safe to write to if it is not actually used by the UI.

Answer 2

On each frame, you are

• creating a Bitmap
• encoding it into a MemoryStream
• creating a BitmapImage
• decoding the MemoryStream into the BitmapImage

Better create a WritableBitmap once, and repeatedly call its WritePixels method.

You may still need to convert the raw buffer, since WPF does not seem to have an equivalent for PixelFormat.Format32bppRgb - or it is perhaps PixelFormats.Bgr32.

var wb = LastFrame as WriteableBitmap;

if (wb == null)
{
    wb = new WriteableBitmap(
        grabResult.Width, grabResult.Height,
        96, 96, PixelFormats.Bgr32, null);
    LastFrame = wb;
}

wb.WritePixels(...);

Answer 3

I am in a similar situation: Pulling live images off a camera and dumping them to the UI for "live" view. I spent a good deal of time trying to find the most efficient solution. For me, the turned out to be BitmapSource.Create. I take the raw array of bytes (plus a structure describing image characteristics like width, height, etc) and use one function call to convert it to a BitmapSource.

Now in my case, the images are greyscale, 8-bit images so if you're trying to show colors, your arguments would be different. But here's a snippet of what I do.

public class XimeaCameraImage : ICameraImage
{
    public unsafe XimeaCameraImage(byte[] imageData, ImgParams imageParams)
    {
        Data             = imageData;
        var fmt          = PixelFormats.Gray8;
        var width        = imageParams.GetWidth();
        var bitsPerPixel = 8;   // TODO: Get ready for other image formats
        var height       = imageParams.GetHeight();
        var stride       = (((bitsPerPixel * width) + 31) / 32) * 4;
        var dpi          = 96.0;

        // Copy the raw, unmanaged, image data from the Sdk.Image object.

        Source = BitmapSource.Create(
            width,
            height,
            dpi,
            dpi,
            fmt,
            BitmapPalettes.Gray256,
            imageData,
            stride);

        Source.Freeze();
    }

    public           byte[]        Data { get; }
    public           BitmapSource  Source            { get; }
}