How to process 10-bit bayer data (bytes array)

Question

I've been stuck for hours on something, I hope you can help me.

I am working on integrating a camera into an application, and I would like to exploit the RAW data that can be retrieved by assigning the value 'bayer-mipi-10grbg' to the 'picture-format' parameter. With my Redmi Note 6, in the OnPictureTaken callback, I get a byte array (15,482,880 bytes).

I have not yet managed to exploit this bytes array.

I read that it was necessary to debayer the data in order to obtain a usable RGB image. So I opted for OpenCv to make my life easier, see the following piece of code:

public void OnPictureTaken(byte[] data, Camera camera)
{
    var height  = 4032;
    var width   = 3024;
    var mBayer  = new Mat(width, height, CvType.Cv16u);
    var mRgb    = new Mat();

    mBayer.Put(0, 0, /* data' */);

    Imgproc.CvtColor(mBayer, mRgb, Imgproc.COLORBayerGR2BGR);

    Imgcodecs.Imwrite($"{_galleryFolder}/{DateTime.Now.ToString("yyMMddhhmmss")}", mRgb);
}

My problem is that to build a Cv16u matrix I can't give it a bytes array to eat, if I understand correctly it needs a shorts array.

My question is, what transformation should I apply to my bytes array to get a valid shorts array?

Edit: The shorts array will obviously not be a simple copy of the bytes array, I imagine for example that the first x bytes correspond to the first value of the shorts array (this is an example). It is this transformation that I do not know about.

Edit2: 4032x3024 is the size I get from the parameter 'raw-size'. I clearly have difficulty making the link between this size and the size of the bytes array I receive. I have uploaded a file that contains the bytes array I receive (saved as is, without processing) : https://ufile.io/gsis9

See https://stackoverflow.com/questions/11112216/how-to-convert-a-byte-array-into-an-int-array just replace `int` with `short` — X39, Feb 04 '19 at 10:31
@x39 It's not that simple. I updated my question. (by the way, there is no question of int here). — Jeremy A., Feb 04 '19 at 10:39
It looks like it's packed (5 bytes represent 4 pixels), but it doesn't exactly add up with the dimensions you provide... could it perhaps be a 4096x3024, because then the total size matches (4096x3024x1.25 = 15,482,880). Could you perhaps upload a sample file we could work with? Could even be lower resolution... — Dan Mašek, Feb 04 '19 at 14:46
@DanMašek Thank you! I have uploaded a file that contains the bytes array as I receive it. I edited my post. — Jeremy A., Feb 04 '19 at 16:06
@JeremyA. A quick look at the data in hex editor shows me that starting at `0xe88f00` (15240960) there's just a bunch of zeros. If we ignore those zeros, the total bytes used works out to be exactly `4032*3024*1.25`. That means that for some reason there are 64 lines of padding at the end of the file which we can safely ignore. — Dan Mašek, Feb 04 '19 at 18:22
@JeremyA. [This](https://developer.xamarin.com/api/field/Android.Graphics.ImageFormat.Raw10/) may describe the packing format. So far the best I could get out is [this image](https://i.imgur.com/WPp5Afy.jpg)... and it's kinda hard to judge whether that's right. — Dan Mašek, Feb 04 '19 at 19:14
@DanMašek You already get better results than me (it's psychedelic this ghost effect). Note that I took this picture in portrait mode, it is as if the height and width were switched (and that a part of the image was missing, at the top and bottom of the keyboard). — Jeremy A., Feb 04 '19 at 19:40
@JeremyA. Good catch with the width/height switch. How about [this](https://i.imgur.com/XtSPGyY.jpg)? [Python prototype](https://pastebin.com/AjXPGrGP) — Dan Mašek, Feb 04 '19 at 19:50
So far it's a trivial approach... read file in chunks of 5 bytes at a time. Unpack each chunk into 4 pixels (based on the above mentioned format). Add those pixels to an array and repeat until you're processed width * height pixels (we ignore any leftover data). Turn it into a single channel `Mat` with 16bit depth, reshape the mat to the proper shape. Run color conversion. | I'll see if I can remember enough Java to give you a proper answer, if you come up with something meanwhile, feel free to self-answer. — Dan Mašek, Feb 04 '19 at 19:57
@DanMašek Yeah, that's it (except for white balance) Well done! Thank you for your precious help. I don't have the project at hand (it will have to wait until tomorrow at work) but I can' t wait to test it. — Jeremy A., Feb 04 '19 at 20:03

score 0 · Answer 1 · answered Feb 05 '19 at 10:26

Ok, so thanks to Dan's precious help, I managed to generate an image from my raw data. A piece of code:

var size    = width * height * 1.25f; // 10-bit depth
var pData   = new List<short>();

for (int i = 0; i < size; i += 5)
{
    pData.Add((short)(data[i] << 2 | ((data[i + 4] >> 0) & 3)));
    pData.Add((short)(data[i + 1] << 2 | ((data[i + 4] >> 2) & 3)));
    pData.Add((short)(data[i + 2] << 2 | ((data[i + 4] >> 4) & 3)));
    pData.Add((short)(data[i + 3] << 2 | ((data[i + 4] >> 6) & 3)));
}

var mBayer = new Mat(width, height, CvType.Cv16u);
mBayer.Put(0, 0, pData.ToArray());

var mRgb = new Mat(width, height, CvType.Cv8u);
Imgproc.CvtColor(mBayer, mRgb, Imgproc.COLORBayerGR2RGB);

This gives me a very greenish image (my next challenge is to manage this green halo).

Bayer works with a pattern that may differ in where the colour components are located. I [wrote some c# code on a previous question here](https://stackoverflow.com/a/50038800/395685) to decode it. It might be usable if you just change the input from `Byte[]` to `Int16[]`... though the final values will of course have to be reduced from 10-bit to 8-bit. You said the format identified itself as "grbg"; that's most likely a square of "green-red | blue-green". This order is important. — Nyerguds, Mar 01 '19 at 08:49

How to process 10-bit bayer data (bytes array)

1 Answers1