1

I am using MagicLeap Headset and MLCamera API to capture a rawvideocapture which the output is YUV_420_888 which I am assuming is YUV420P. API returns yBuffer, uBuffer and vBuffer separately. I am having trouble combining these channels on c# without bitmap since I am using unity I am using Mono. What I am trying to do is to combine these channels and send it to my remote python server to process the image that I have captured. To process the image, it needs to be a full image. I have tried just using the Y plane and creating a gray-scale image but the server couldn't process it so I need to combine all 3 channels on the client and then compress it to preferable jpeg since the size decreases drastically and I am processing the images at 420x420 size although the camera output is 1920x1080. I am trying different methods for the last week and half but couldn't find something decent. There are a couple methods especially for Android but I don't want to convert it to NV21 if I don't have to. I have also seen one with ARCore but I also can't use that one since I am using MagicLeap.

PS: The latency and the processing time is super important so if there is a way to convert YCbCr to jpeg directly without converting it to RGB, I think it would help my case better but I don't know if it's possible. In general I think I lack some basic knowledge that prevents me from going further.

Any help is greatly appreciated!

Berk
  • 11
  • 1
  • Have you tried creating a texture out of it using the yuv space `Texture2D tex = new Texture2D(420, 420, TextureFormat.YUY2, false);` and then converting that texture to a jpg using `ImageConversion.EncodeToJPG(Texture2D tex);`? – Remy Jun 17 '20 at 14:20
  • Hello Remy thanks for suggestion, I haven't tried it because I am not sure how to convert YUV_420_888 to YUY2. Do you think it is going to be faster than other possible solutions? – Berk Jun 17 '20 at 14:28

1 Answers1

1

I've tried something similar in the past, was beating my head on the YUV420 stuff for weeks, but couldn't solve it. In the end, I bought this library OpenCV for Unity. It has custom parts just for the MagicLeap, including reading frames from the Camera in reduced resolution for speed up. I'm not sure how ever if it managed real time. Maybe in the reduced resolution, yes.

Cornelius Einem
  • 60
  • 1
  • 1
  • 6