Scaling YUV420 Image using libyuv produces weird output

Question

Possible duplicate of This question with major parts picked from here. I've tried whatever solutions were provided there, they don't work for me.

Background

I'm capturing an image in YUV_420_888 image format returned from ARCore's frame.acquireCameraImage() method. Since I've set the camera configuration at 1920*1080 resolution, I need to scale it down to 224*224 to pass it to my tensorflow-lite implementation. I do that by using LibYuv library through the Android NDK.

Implementation

Prepare the image frames

    //Figure out the source image dimensions
    int y_size = srcWidth * srcHeight;

    //Get dimensions of the desired output image
    int out_size = destWidth * destHeight;

    //Generate input frame
    i420_input_frame.width = srcWidth;
    i420_input_frame.height = srcHeight;
    i420_input_frame.data = (uint8_t*) yuvArray;
    i420_input_frame.y = i420_input_frame.data;
    i420_input_frame.u = i420_input_frame.y + y_size;
    i420_input_frame.v = i420_input_frame.u + (y_size / 4);

    //Generate output frame
    free(i420_output_frame.data);
    i420_output_frame.width = destWidth;
    i420_output_frame.height = destHeight;
    i420_output_frame.data = new unsigned char[out_size * 3 / 2];
    i420_output_frame.y = i420_output_frame.data;
    i420_output_frame.u = i420_output_frame.y + out_size;
    i420_output_frame.v = i420_output_frame.u + (out_size / 4);

I scale my image using Libyuv's I420Scale method

libyuv::FilterMode mode = libyuv::FilterModeEnum::kFilterBox;
jint result = libyuv::I420Scale(i420_input_frame.y, i420_input_frame.width,
                                i420_input_frame.u, i420_input_frame.width / 2,
                                i420_input_frame.v, i420_input_frame.width / 2,
                                i420_input_frame.width, i420_input_frame.height,
                                i420_output_frame.y, i420_output_frame.width,
                                i420_output_frame.u, i420_output_frame.width / 2,
                                i420_output_frame.v, i420_output_frame.width / 2,
                                i420_output_frame.width, i420_output_frame.height,
                                mode);

and return it to java

    //Create a new byte array to return to the caller in Java
    jbyteArray outputArray = env -> NewByteArray(out_size * 3 / 2);
    env -> SetByteArrayRegion(outputArray, 0, out_size, (jbyte*) i420_output_frame.y);
    env -> SetByteArrayRegion(outputArray, out_size, out_size / 4, (jbyte*) i420_output_frame.u);
    env -> SetByteArrayRegion(outputArray, out_size + (out_size / 4), out_size / 4, (jbyte*) i420_output_frame.v);

Actual image :

What it looks like post scaling :

What it looks like if I create an Image from the i420_input_frame without scaling :

Since the scaling messes up the colors big time, tensorflow fails to recognize objects properly. (It recognizes properly in their sample application) What am I doing wrong to mess up the colors big time?

Answer 1

Either I was doing something wrong (Which I couldn't fix) or LibYuv does not handle colors properly while dealing with YUV images from Android.

Refer official bug posted on Libyuv library : https://bugs.chromium.org/p/libyuv/issues/detail?id=815&can=1&q=&sort=-id

They suggested I use a method Android420ToI420() first and then I apply whatever transformations I need. I ended up using Android420ToI420() first, then Scaling, then transformation to RGB. In the end, the output was slightly better than the cup image posted above but the distorted colors were still present. I ended up using OpenCV to shrink the image and convert it to RGBA or RGB formats.

// The camera image received is in YUV YCbCr Format at preview dimensions
// so we will scale it down to 224x224 size using OpenCV
// Y plane (0) non-interleaved => stride == 1; U/V plane interleaved => stride == 2
// Refer : https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888
val cameraPlaneY = cameraImage.planes[0].buffer
val cameraPlaneUV = cameraImage.planes[1].buffer

// Create a new Mat with OpenCV. One for each plane - Y and UV
val y_mat = Mat(cameraImage.height, cameraImage.width, CvType.CV_8UC1, cameraPlaneY)
val uv_mat = Mat(cameraImage.height / 2, cameraImage.width / 2, CvType.CV_8UC2, cameraPlaneUV)
var mat224 = Mat()
var cvFrameRGBA = Mat()

// Retrieve an RGBA frame from the produced YUV
Imgproc.cvtColorTwoPlane(y_mat, uv_mat, cvFrameRGBA, Imgproc.COLOR_YUV2BGRA_NV21)


//Then use this frame to retrieve all RGB channel data
//Iterate over all pixels and retrieve information of RGB channels
  for(rows in 1 until cvFrameRGBA.rows())
      for(cols in 1 until cvFrameRGBA.cols()) {
          val imageData = cvFrameRGBA.get(rows, cols)
          // Type of Mat is 24
          // Channels is 4
          // Depth is 0
          rgbBytes.put(imageData[0].toByte())
          rgbBytes.put(imageData[1].toByte())
          rgbBytes.put(imageData[2].toByte())
      }

Answer 2

The color problem is caused because you are working with a different YUV format. The YUV format that camera frameworks use is YUV NV21. This format (NV21) is the standard picture format on Android camera preview. YUV 4:2:0 planar image, with 8 bit Y samples, followed by interleaved V/U plane with 8bit 2x2 subsampled chroma samples.

If your colors are inversed, it means that:

• You are working with a YUV NV12 (plane U is V and V is U).
• One of your color planes is doing something weird.

To work properly with libyuv I suggest you to convert your camera output to a YUV I420 using transformI420 method and sending the format by parameter:

return libyuv::ConvertToI420(src, srcSize, //src data
                             dstY, dstWidth, //dst planes
                             dstU, dstWidth / 2,
                             dstV, dstWidth / 2,
                             cropLeft, cropTop, //crop start
                             srcWidth, srcHeight, //src dimensions
                             cropRight - cropLeft, cropBottom - cropTop, //dst dimensions
                             rotationMode,
                             libyuv::FOURCC_NV21); //libyuv::FOURCC_NV12

After do this conversion you will be able to properly work with libyuv using all the I420scale, I420rotate... and so on. Your scale method should look like:

JNIEXPORT jint JNICALL
Java_com_aa_project_images_yuv_myJNIcl_scaleI420(JNIEnv *env, jclass type,
                                                 jobject srcBufferY,
                                                 jobject srcBufferU,
                                                 jobject srcBufferV,
                                                 jint srcWidth, jint srcHeight,
                                                 jobject dstBufferY,
                                                 jobject dstBufferU,
                                                 jobject dstBufferV,
                                                 jint dstWidth, jint dstHeight,
                                                 jint filterMode) {

    const uint8_t *srcY = static_cast<uint8_t *>(env->GetDirectBufferAddress(srcBufferY));
    const uint8_t *srcU = static_cast<uint8_t *>(env->GetDirectBufferAddress(srcBufferU));
    const uint8_t *srcV = static_cast<uint8_t *>(env->GetDirectBufferAddress(srcBufferV));
    uint8_t *dstY = static_cast<uint8_t *>(env->GetDirectBufferAddress(dstBufferY));
    uint8_t *dstU = static_cast<uint8_t *>(env->GetDirectBufferAddress(dstBufferU));
    uint8_t *dstV = static_cast<uint8_t *>(env->GetDirectBufferAddress(dstBufferV));

    return libyuv::I420Scale(srcY, srcWidth,
                             srcU, srcWidth / 2,
                             srcV, srcWidth / 2,
                             srcWidth, srcHeight,
                             dstY, dstWidth,
                             dstU, dstWidth / 2,
                             dstV, dstWidth / 2,
                             dstWidth, dstHeight,
                             static_cast<libyuv::FilterMode>(filterMode));
}

If you want to convert this image to a JPEG after all the process. You can use I420toNV21 method and after that use the android native conversion from YUV to JPEG. Also you can use libJpegTurbo which is a complementary library for this kind of situations.