0

I am working with mediaframes (Kinect) to get colour, Depth/Infrared frames on UWP in realtime. This is to store frame data on disk and later process it.

For colour, I get pixels in bytes by using Memorystream.

                // Get the Individual color Frame
            var vidFrame = clrFrame?.VideoMediaFrame;
            {
                if (vidFrame == null) return;

                // create a UWP SoftwareBitmap and copy Color Frame into Bitmap
                SoftwareBitmap sbt = new SoftwareBitmap(vidFrame.SoftwareBitmap.BitmapPixelFormat, vidFrame.SoftwareBitmap.PixelWidth, vidFrame.SoftwareBitmap.PixelHeight);
                vidFrame.SoftwareBitmap.CopyTo(sbt);

                // PixelFormat needs to be in 8bit for Colour only
                if (sbt.BitmapPixelFormat != BitmapPixelFormat.Bgra8)
                    sbt = SoftwareBitmap.Convert(vidFrame.SoftwareBitmap, BitmapPixelFormat.Bgra8);

                if (source != null)
                {
                    var ignore = Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, async () =>
                    {
                        
                        extBitmap = new WriteableBitmap(sbt.PixelWidth, sbt.PixelHeight);
                        sbt.CopyToBuffer(extBitmap.PixelBuffer);
                        byte[] pixels = PixelBufferToWriteableBitmap(extBitmap);                       

                        extBitmap.Invalidate();
                        await SavePixelsToFile(pixels);
                    });
                }
            }


public async Task<byte[]> PixelBufferToWriteableBitmap(WriteableBitmap wb)
    {
        using (Stream stream = wb.PixelBuffer.AsStream())
        {
            using (MemoryStream memoryStream = new MemoryStream())
            {
                await stream.CopyToAsync(memoryStream);
                byte[] pixels = memoryStream.ToArray();
                return pixels;
            }
        }
    }

The Infrared pixelformat is Gray16 (in SoftwareBitmap); I want to keep the raw pixel data (so no data is lost from the frame) and write it to the localfolder in ushort[] array.

Following are the links, I came across on how to get set pixel from software bitmap. However, it is bgra to byte and I want to convert software bitmap into ushort.

How to set/get pixel from Softwarebitmap

https://learn.microsoft.com/en-us/windows/uwp/audio-video-camera/imaging

I am new at this and not sure how to proceed, with this.

Can someone please help?

EDIT

I figured that a buffer mediaframe can be converted to byte array by doing the following:

public async Task<byte[]> BufferStreamTobyte(BufferMediaFrame buffFrame) {

        using (Stream stream = buffFrame.Buffer.AsStream())
        {
            using (MemoryStream memoryStream = new MemoryStream())
            {
                await stream.CopyToAsync(memoryStream);
                byte[] pixels = memoryStream.ToArray();
                return pixels;
            }
        }
    }

But I am not sure if I loose information of Infrared frame by doing this. Since Infrared and Depth are 16 bits per pixel and this current byte conversion holds 8 bpp. For this ushort[] would be able to hold 16bpp. I very new to this and not sure so I hope I have gotten this right?

EDIT 2:

I have got the pixel data in byte[]. I understand that byte is 8 bits and short is 16 bits so I changed the length of the arrays:

int width = softwareBitmap.PixelWidth;
int height = softwareBitmap.PixelHeight;
int length = width * height * 2;

byte[] irbyteData = new byte[length]; // *2 to get 16 bit

var irshortData = new ushort[width * height]; // 16 bit ushort 

IntPtr ptr = (IntPtr)pixelBytesAddress;
Marshal.Copy(ptr, irbyteData, 0, length); //seems to successfully get pixels from memory but not sure if this is lossless
Peri
  • 11
  • 3
  • Your question is unclear. You state that _"the pixel format is Gray16"_ and that you _"want to keep the raw pixel data"_, but the code you posted assumes the pixel format is 32-bit RGBA. And it doesn't even read a buffer anyway; it just generates a grayscale gradient based on the X value of the pixel within the bitmap. In what way is the code you posted relevant? Where is the _actual_ code that does something similar to what you want to happen? – Peter Duniho Sep 30 '20 at 17:05
  • The pixel format is Gray16 when the frame arrives but how do I get the pixels in ushort[] (16bit) to store this information on disk? – Peri Oct 01 '20 at 14:21
  • so, you just want to same Gray16 frame as image into disk right ? – Nico Zhu Oct 02 '20 at 07:20
  • Yes, I want to save Gray16 frame to disk without losing any information. – Peri Oct 02 '20 at 12:14

2 Answers2

0

I would try using PNGs

ex:

    using(var fs=new FileStream("sample.png"))
{
        BitmapSource bmpSource=BitmapSource(Width,Height,96,96,PixelFormats.Gray16,
        BitmapPalettes.Gray16,buffer,ImageWidth*sizeof(ushort));
    PngBitmapEncoder enc = new PngBitmapEncoder();
    end.Frames.Add(BitmapFrame.Create(bmpSource));
    enc.Save(fs)
}
  1. Sorry if there's a typo in a code I write from memory on a computer without IDE
  2. Add System.Windows.media.Imaging to your usings
  3. buffer is a ushort array with the pixel values (should be in total length: width*height*2)
Felix
  • 109
  • 2
  • 4
  • Thank you Felix for the solution. I will give it a go. Will this ensure that the data is not lost and is stored for further processing? – Peri Nov 16 '20 at 14:20
  • When I check the output file with MediaInfo (v18.05) it reports:16 bit per pixel, **lossless** compression so it appears so, it may vary on different OS though – Felix Nov 18 '20 at 09:22
  • BitmapSource implementation is quite different in UWP. https://learn.microsoft.com/en-us/uwp/api/windows.ui.xaml.media.imaging.bitmapsource?view=winrt-19041. The code does not seem to work. I have tried other ways around it but does not seem to work. – Peri Nov 20 '20 at 13:18
0

I found a solution to copy IntPtr data to ushort array through the following link:

Copy from IntPtr (16 bit) array to managed ushort

I get the address IntPtr by

using (var input = softwareBitmap.LockBuffer(BitmapBufferAccessMode.ReadWrite))
using (var inputReference = input.CreateReference())    
((IMemoryBufferByteAccess)inputReference).GetBuffer(out inputBytes, out inputCapacity);

IntPtr ptr = (IntPtr)inputBytes;
Marshal.Copy(ptr, infraredbyteData, 0, length);

I get the bytes with length( width *height *2 ) to hold 16 bit data.

Later I convert it to ushort by

var size = infraredbyteData.Length / 2; 
ushort[] output = new ushort[size]; // every ushort is 2 bytes
Buffer.BlockCopy(infraredbyteData, 0, output, 0, infraredbyteData.Length);

This seems to work!

Peri
  • 11
  • 3