Colour/chroma texture is stretched when converting from YUV-NV12 to RGB using OpenGL ES

Question

I have a byte buffer of YUV-NV12 formatted image data. When I try to convert it to RGB, I get an output with a stretched colour (chroma) layer like in the image below.

I followed this great answer, which guides to convert YUV-NV21 to RGB. Since NV-12 is just NV-21 with flipped U and V data, the only change I should do is to replace u and v values in the fragment shader.

Vertex shader:

precision mediump float;
uniform mat4 uMVPMatrix;
attribute vec4 vPosition;
attribute vec4 vTextureCoordinate;
varying vec2 position;

void main()
{
   gl_Position = uMVPMatrix * vPosition;
   position = vTextureCoordinate.xy;
}

Fragment shader:

precision mediump float;
varying vec2 position;
uniform sampler2D uTextureY;
uniform sampler2D uTextureUV;

void main() 
{
    float y, u, v;
    y = texture2D(uTextureY, position).r;
    u = texture2D(uTextureUV, position).a - 0.5;
    v = texture2D(uTextureUV, position).r - 0.5;

    float r, g, b;
    r = y + 1.13983 * v;
    g = y - 0.39465 * u - 0.58060 * v;
    b = y + 2.03211 * u;

    gl_FragColor = vec4(r, g, b, 1.0);
}

Split and put image data into 2 ByteBuffer's which are mYBuffer and mUVBuffer. mSourceImage is just a Buffer which contains the image data as byte data.

ByteBuffer bb = (ByteBuffer) mSourceImage;
if (bb == null) {
    return;
}
int size = mWidth * mHeight;

bb.position(0).limit(size);
mYBuffer = bb.slice();
bb.position(size).limit(bb.remaining());
mUVBuffer = bb.slice();

Generating textures:

GLES20.glGenTextures(2, mTexture, 0);

for(int i = 0; i < 2; i++) {
    GLES20.glActiveTexture(GLES20.GL_TEXTURE0 + i);
    GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTexture[i]);

    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
    GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
}

Passing buffer data to textures:

mTextureYHandle = GLES20.glGetUniformLocation(mProgramId, "uTextureY");
mTextureUVHandle = GLES20.glGetUniformLocation(mProgramId, "uTextureUV");

GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTexture[0]);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, mWidth, mHeight, 0, GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, mYBuffer);
GLES20.glUniform1i(mTextureYHandle, 0);

GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTexture[1]);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE_ALPHA, mWidth / 2, mHeight / 2, 0, GLES20.GL_LUMINANCE_ALPHA, GLES20.GL_UNSIGNED_BYTE, mUVBuffer);
GLES20.glUniform1i(mTextureUVHandle, 1);

I couldn't figure out why I'm getting such an output. Any help would be much appreciated.

Amila Abeygunasekara · Answer 1 · 2020-10-30T04:24:53.733

Nevermind, It was a tiny mistake in my code.

When splitting the byte buffer, I have used bb.position(size).limit(bb.remaining()) for the UV buffer. For some reason, bb.remaining() become 0 after getting some frames (This is actually a camera preview). Therefore I have changed it to bb.position(size).limit(size + size / 2).

Also the assumption I made by reading this,

the only change I should do is to replace u and v values in the fragment shader

appears to be wrong. It is observed that GL20.GL_LUMINANCE_ALPHA will always put the U byte into the A component of the texture, and the V byte into R, G, B components (You can use either one). Hence, no need to swap u and v values in the fragment shader (I have edited my question with the correct fragment shader code).

I will keep the question hoping this would help someone in the future.

I know why `bb.remaining()` become 0. Because `remaining()` returns the number of elements between the current position and the limit. The distance between `position(size)` and `limit(size)` is 0. Setting the code as `bb.position(size).limit(size + size / 2)` is correct. — zeitgeist, Mar 29 '22 at 08:09

Colour/chroma texture is stretched when converting from YUV-NV12 to RGB using OpenGL ES

1 Answers1