YUV420p to RGB conversion has shifted U and V values - iOS Unity3D

Question

I have a native plugin in Unity that decodes H264 frames to YUV420p using FFMPEG.

To display the output image, I rearrange the YUV values into an RGBA texture and convert YUV to RGB using Unity shader (just to make it faster).

The following is the rearrangement code in my native plugin:

unsigned char* yStartLocation = (unsigned char*)m_pFrame->data[0];
unsigned char* uStartLocation = (unsigned char*)m_pFrame->data[1];
unsigned char* vStartLocation = (unsigned char*)m_pFrame->data[2];

for (int y = 0; y < height; y++)
{
    for (int x = 0; x < width; x++)
    {

        unsigned char* y = yStartLocation + ((y * width) + x);
        unsigned char* u = uStartLocation + ((y * (width / 4)) + (x / 2));
        unsigned char* v = vStartLocation + ((y * (width / 4)) + (x / 2));
        //REF: https://en.wikipedia.org/wiki/YUV

        // Write the texture pixel
        dst[0] = y[0]; //R
        dst[1] = u[0]; //G
        dst[2] = v[0]; //B
        dst[3] = 255;  //A

        // To next pixel
        dst += 4;

        // dst is the pointer to target texture RGBA data
    }
}

The shader that converts YUV to RGB works perfectly and I've used it in multiple projects.

Now, I'm using the same code to decode on iOS platform. But for some reason the U and V values are now shifted:

Y Texture

U Texture

Is there anything that I'm missing for iOS or OpenGL specifically?

Any help greatly appreciated. Thank You!

Please note that I filled R=G=B = Y for the first screenshot and U for the second.(if that makes sense)

Edit: Heres the output that I'm getting:

Edit2: Based on some research I think it may have something to do with Interlacing.

ref: Link

For now I've shifted to CPU based YUV-RGB conversion using sws_scale and it works fine.

Have you tried [this](https://stackoverflow.com/a/16421553/3785314)? — Programmer, Aug 24 '18 at 21:18
@Programmer YUYV (in the answer you suggested) is packed YUV 4:2:2, way different than the OP YUV420p, that is planar YUV 4:2:0 — roalz, Aug 24 '18 at 21:45
Shooting in the dark here (no knowledge of iOS platform) but... are you sure it's not interpreting the RGBA as ... let's say, RBGA instead? What if you invert u[0] and v[0] in the assignments to dst? — roalz, Aug 24 '18 at 21:54
Not answering your question, but you should know that this step of building `dst` is time consuming and not necessary. You can have a shader that takes a YUV420 texture as input, and results in a RGB. — Alex Cohn, Aug 25 '18 at 09:47
So in your second picture, the U channel looks wrong, isn't it? Can you post the correct expected picture? — Alex Cohn, Aug 25 '18 at 09:49
I'm pretty sure the decoded output from FFMPEG is YUV420p. Do you know any way of knowing the output format? @Programmer — rohit n, Aug 25 '18 at 22:38
@AlexCohn I'll post the expected output asap. But, hoelw do you actually take the AVFrame and send it to Unity as a Texture without building the texture first? Or do you mean I don't need the redundant data ? — rohit n, Aug 25 '18 at 22:40
The best service to analyze YUV images I know is http://rawpixels.net/ — Alex Cohn, Aug 26 '18 at 08:26
It's not only that you inflate U and V pixel data, or pass redundant fourth byte. Your code must copy all pixels from the YUV frame to `dst`, which itself costs significant time. [Here](https://hub.jmonkeyengine.org/t/solved-rendering-a-yuv420-video-data-need-help-writing-the-material/40026/6) is just one example how you can pass the original frame (they use 3 separate textures) to fragment shader that combines them together into RGB. It is possible to use a single texture of size Wx(H*3/2), but I am not sure if this approach has performance advantages over the tri-texture above. — Alex Cohn, Aug 26 '18 at 08:35
@roalz I tested the app with different channels exchanged just to see if I misread the values. But, thats not the case. I've edited the question with the current output I'm getting. — rohit n, Aug 27 '18 at 14:49

score 1 · Accepted Answer · answered Sep 28 '18 at 17:28

The problem was on this line :

uStartLocation + ((y * (width / 4)) + (x / 2));

It should be

uStartLocation + (((y / 2) * (width / 2)) + (x / 2));

Since int rounding was causing the whole frame to shift. Very silly error trying to optimize calculations.

Hope it helps someone.

YUV420p to RGB conversion has shifted U and V values - iOS Unity3D

1 Answers1