Convert YUV420 to RGB using OpenCVSharp

Question

I want to convert a YUV stream into RGB bytes such that they can be displayed through a WPF Image.

All YUV values are placed in their respective arrays per frame y, u, v. chroma width and height are half of the respective luma dimensions.

byte[] y = (byte[])yData;
byte[] u = (byte[])uData;
byte[] v = (byte[])vData;

var ym = new Mat(new[] { lumaHeight, lumaWidth }, MatType.CV_8UC1, y, new long[] { lumaStride });
var um = new Mat(new[] { chromaWidth, chromaHeight }, MatType.CV_8UC1, u, new long[] { chromaStride});
var vm = new Mat(new[] { chromaWidth, chromaHeight }, MatType.CV_8UC1, v, new long[] { chromaStride});

I use the following code to pass the data to openCV:

var combinedSource = new[] { ym, um, vm };

var m = new Mat();
var src = InputArray.Create(combinedSource);
var @out = OutputArray.Create(m);
Cv2.CvtColor(src, @out, ColorConversionCodes.YUV2BGR);
ImageData = @out.GetMat().ToBytes();

But I receive the error: {"!_src.empty()"} The yuv arrays are definitely not empty.

I try another method using:

var combinedOut = new Mat(new[] { lumaHeight, lumaWidth }, MatType.CV_8UC3);
Cv2.Merge(combinedSource, combinedOut);
var bgra = combinedOut.CvtColor(ColorConversionCodes.YUV2BGR);
ImageData = bgra.ToBytes();

But receive the error {"mv[i].size == mv[0].size && mv[i].depth() == depth"}

Why do I receive these errors and what is the correct way to convert?

score 0 · Accepted Answer · answered Jul 03 '19 at 07:16

Following this post I was able to come to the following solution:

    private byte[] YuvToRgbOpenCv(object luma, object chroma, object yData, object uData, object vData)
    {
        int[] lumaArray = (int[])luma;
        int[] chromaArray = (int[])chroma;

        int lumaWidth = lumaArray[0];
        int lumaHeight = lumaArray[1];

        int chromaWidth = chromaArray[0];
        int chromaHeight = chromaArray[1];

        byte[] y = (byte[])yData;
        byte[] u = (byte[])uData;
        byte[] v = (byte[])vData;

        var ym = new Mat(new[] { lumaHeight, lumaWidth }, MatType.CV_8UC1, y);
        var um = new Mat(new[] { chromaHeight, chromaWidth }, MatType.CV_8UC1, u);
        var vm = new Mat(new[] { chromaHeight, chromaWidth }, MatType.CV_8UC1, v);

        var umResized = um.Resize(new OpenCvSharp.Size(lumaWidth, lumaHeight), 0, 0, InterpolationFlags.Nearest);
        var vmResized = vm.Resize(new OpenCvSharp.Size(lumaWidth, lumaHeight), 0, 0, InterpolationFlags.Nearest);

        var yuvMat = new Mat();
        var resizedChannels = new[] { ym, umResized, vmResized };

        Cv2.Merge(resizedChannels, yuvMat);

        var bgr = yuvMat.CvtColor(ColorConversionCodes.YUV2BGR);

        var result = bgr.ToBytes();

        return result;
    }

I had to resize the U, V data length to match the length of Y.

score -1 · Answer 2 · answered Jul 02 '19 at 10:14

Why don't you try to convert by looping through all array of source and output array. For this method to work you have to combine all arrays into one and then pass that array to the following function

private static unsafe void YUV2RGBManaged(byte[] YUVData, byte[] RGBData, int width, int height)
    {
        fixed(byte* pRGBs = RGBData, pYUVs = YUVData)
        {
            for (int r = 0; r < height; r++)
            {
                byte* pRGB = pRGBs + r * width * 3;
                byte* pYUV = pYUVs + r * width * 2;

                //process two pixels at a time
                for (int c = 0; c < width; c += 2)
                {
                    int C1 = pYUV[1] - 16;
                    int C2 = pYUV[3] - 16;
                    int D = pYUV[2] - 128;
                    int E = pYUV[0] - 128;

                    int R1 = (298 * C1 + 409 * E + 128) >> 8;
                    int G1 = (298 * C1 - 100 * D - 208 * E + 128) >> 8;
                    int B1 = (298 * C1 + 516 * D + 128) >> 8;

                    int R2 = (298 * C2 + 409 * E + 128) >> 8;
                    int G2 = (298 * C2 - 100 * D - 208 * E + 128) >> 8;
                    int B2 = (298 * C2 + 516 * D + 128) >> 8;
#if true
                    //check for overflow
                    //unsurprisingly this takes the bulk of the time.
                    pRGB[0] = (byte)(R1 < 0 ? 0 : R1 > 255 ? 255 : R1);
                    pRGB[1] = (byte)(G1 < 0 ? 0 : G1 > 255 ? 255 : G1);
                    pRGB[2] = (byte)(B1 < 0 ? 0 : B1 > 255 ? 255 : B1);

                    pRGB[3] = (byte)(R2 < 0 ? 0 : R2 > 255 ? 255 : R2);
                    pRGB[4] = (byte)(G2 < 0 ? 0 : G2 > 255 ? 255 : G2);
                    pRGB[5] = (byte)(B2 < 0 ? 0 : B2 > 255 ? 255 : B2);
#else
                    pRGB[0] = (byte)(R1);
                    pRGB[1] = (byte)(G1);
                    pRGB[2] = (byte)(B1);

                    pRGB[3] = (byte)(R2);
                    pRGB[4] = (byte)(G2);
                    pRGB[5] = (byte)(B2);
#endif

                    pRGB += 6;
                    pYUV += 4;
                }
            }
        }
    }

I see this code is a copy of [another answer](https://stackoverflow.com/a/16108293/873443). While converting my YUV420 input to YUYV format, which I believe this code uses, is a possible route, I'd prefer to put my faith in OpenCV which is most likely more reliable and faster than this method. On a side note, this would better go in a comment than as an answer. — FinalFortune, Jul 03 '19 at 07:06

Convert YUV420 to RGB using OpenCVSharp

2 Answers2