camera2 captured picture - conversion from YUV_420_888 to NV21

Question

Via the camera2 API we are receiving an Image object of the format YUV_420_888. We are using then the following function for conversion to NV21:

private static byte[] YUV_420_888toNV21(Image image) {
    byte[] nv21;
    ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
    ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
    ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();

    int ySize = yBuffer.remaining();
    int uSize = uBuffer.remaining();
    int vSize = vBuffer.remaining();

    nv21 = new byte[ySize + uSize + vSize];

    //U and V are swapped
    yBuffer.get(nv21, 0, ySize);
    vBuffer.get(nv21, ySize, vSize);
    uBuffer.get(nv21, ySize + vSize, uSize);

    return nv21;
}

While this function works fine with cameraCaptureSessions.setRepeatingRequest, we get a segmentation error in further processing (on the JNI side) when calling cameraCaptureSessions.capture. Both request YUV_420_888 format via ImageReader.

How come the result is different for both function calls while the requested type is the same?

Update: As mentioned in the comments I get this behaviour because of different image sizes (much larger dimension for the capture request). But our further processing operations on the JNI side are the same for both requests and don't depend on image dimensions (only on the aspect ratio, which is in both cases the same).

I don't know why the two situations produce different results. But you can find out. Compare the **Image** parameters that you receive in two cases. Pay attention to raw strides and pixel strides, especially for U and V planes. — Alex Cohn, Oct 09 '18 at 19:46
One important difference may be that you request different sizes. Not that `capture` is so much bigger, but it can have differnet padding due to this change. — Alex Cohn, Oct 09 '18 at 19:50
@AlexCohn You are completly right, the different behaviour is a result of different image sizes. How would I have to adjust the function to respect a differnt padding? — Alexander Belokon, Oct 10 '18 at 09:47

Alex Cohn · Accepted Answer · 2020-09-20T14:02:58.107

Your code will only return correct NV21 if there is no padding at all, and U and V plains overlap and actually represent interlaced VU values. This happens quite often for preview, but in such case you allocate extra w*h/4 bytes for your array (which presumably is not a problem). Maybe for captured image you need a more robust implemenation, e.g.

private static byte[] YUV_420_888toNV21(Image image) {

    int width = image.getWidth();
    int height = image.getHeight(); 
    int ySize = width*height;
    int uvSize = width*height/4;

    byte[] nv21 = new byte[ySize + uvSize*2];

    ByteBuffer yBuffer = image.getPlanes()[0].getBuffer(); // Y
    ByteBuffer uBuffer = image.getPlanes()[1].getBuffer(); // U
    ByteBuffer vBuffer = image.getPlanes()[2].getBuffer(); // V

    int rowStride = image.getPlanes()[0].getRowStride();
    assert(image.getPlanes()[0].getPixelStride() == 1);

    int pos = 0;

    if (rowStride == width) { // likely
        yBuffer.get(nv21, 0, ySize);
        pos += ySize;
    }
    else {
        long yBufferPos = -rowStride; // not an actual position
        for (; pos<ySize; pos+=width) {
            yBufferPos += rowStride;
            yBuffer.position(yBufferPos);
            yBuffer.get(nv21, pos, width);
        }
    }

    rowStride = image.getPlanes()[2].getRowStride();
    int pixelStride = image.getPlanes()[2].getPixelStride();

    assert(rowStride == image.getPlanes()[1].getRowStride());
    assert(pixelStride == image.getPlanes()[1].getPixelStride());
    
    if (pixelStride == 2 && rowStride == width && uBuffer.get(0) == vBuffer.get(1)) {
        // maybe V an U planes overlap as per NV21, which means vBuffer[1] is alias of uBuffer[0]
        byte savePixel = vBuffer.get(1);
        try {
            vBuffer.put(1, (byte)~savePixel);
            if (uBuffer.get(0) == (byte)~savePixel) {
                vBuffer.put(1, savePixel);
                vBuffer.position(0);
                uBuffer.position(0);
                vBuffer.get(nv21, ySize, 1);
                uBuffer.get(nv21, ySize + 1, uBuffer.remaining());

                return nv21; // shortcut
            }
        }
        catch (ReadOnlyBufferException ex) {
            // unfortunately, we cannot check if vBuffer and uBuffer overlap
        }

        // unfortunately, the check failed. We must save U and V pixel by pixel
        vBuffer.put(1, savePixel);
    }

    // other optimizations could check if (pixelStride == 1) or (pixelStride == 2), 
    // but performance gain would be less significant

    for (int row=0; row<height/2; row++) {
        for (int col=0; col<width/2; col++) {
            int vuPos = col*pixelStride + row*rowStride;
            nv21[pos++] = vBuffer.get(vuPos);
            nv21[pos++] = uBuffer.get(vuPos);
        }
    }

    return nv21;
}

If you anyway intend to pass the resulting array to C++, you can take advantage of the fact that

the buffer returned will always have isDirect return true, so the underlying data could be mapped as a pointer in JNI without doing any copies with GetDirectBufferAddress.

This means that same conversion may be done in C++ with minimal overhead. In C++, you may even find that the actual pixel arrangement is already NV21!

PS Actually, this can be done in Java, with negligible overhead, see the line if (pixelStride == 2 && … above. So, we can bulk copy all chroma bytes to the resulting byte array, which is much faster than running the loops, but still slower than what can be achieved for such case in C++. For full implementation, see Image.toByteArray().

it crashed for me when `yBuffer.position` was out of bounds in the last run. To fix it change the loop to `int bufferPos = yBuffer.position(); for (; pos — uwe, Jun 21 '19 at 10:56
@uwe thanks for sharing, I have updated my answer to address this edge cade — Alex Cohn, Jun 21 '19 at 17:10
I don't understand the socalled "shortcut" in the code. Clearly the second branch will never ever be met: if (uBuffer.get(0) == 0) { vBuffer.put(1, (byte)255); if (uBuffer.get(0) == 255) { } } — Casper Bang, Feb 03 '20 at 15:44
@CasperBang the second branch is chosen pretty often, because the underlying library uses *overlapping* buffers for U and V. So, `uBuffer[0]` and `vBuffer[1]` look at the same byte in memory. In C, we can simply compare two pointers. Not in Java What we can do, is check that the values of `uBuffer[0]` and `vBuffer[1]` are the same even after some manipulation. — Alex Cohn, Feb 03 '20 at 21:59
Hey there! Just saw your updates and I have a question. It seems that every time you call `vBuffer.put()` you call it on the first index with the value `savePixel` which is `vBuffer.get(1)` in the first place. Meaning you're setting it to the value that already exists at that index. So can't all those put calls be removed? Asking because I am getting a ReadOnlyBufferException when trying to `put()` — kjanderson2, Feb 03 '20 at 22:47
@kjanderson2: Thanks for your report! If `vBuffer` is read-only, you cannot prove that `uBuffer` and `vBuffer` actually overlap, so you have to go the safe way, with the loop by **row ** and **col**. `vBuffer.put()` is not to make some persistent change in the buffer, but only to sniff this overlap. Unfortunately, there is no public API to check whether a ByteBuffer is read-only, so the exception is the only way to find out. I have added the relevant `try … catch` to cover your situation gracefully. Note that still it may be possible to perform direct manipulations in C++. — Alex Cohn, Feb 04 '20 at 08:25
I'm curious if I did this correctly. To get around the ReadOnlyException, I copied the contents of the plane buffer to a new buffer: `val vPlaneBuffer = image.planes[2].buffer // image.planes[2].buffer is read only, so we make a copy so we can update the buffer val vBuffer = ByteBuffer.allocate(vPlaneBuffer.capacity()) vBuffer.put(vPlaneBuffer)` Would this work as well when comparing or am I no longer comparing the right information? — Stephen Emery, Feb 04 '20 at 17:14
@StephenEmery No, as soon as you copy the **vBuffer**, you loose its overlap with the **uBuffer** for good. You could, theoretically, simply check that all values in vBuffer at odd offsets (i.e. get(1), get(3), get(5), …) are equal to the values in uBuffer with even offsets (i.e. get(0), get(2), get(4), …). If all `uvSize` pairs are equal, you can use the shortcut, only copying the vBuffer to resulting nv21 array. This isn't practical, though: your gain (if you are lucky) will be lost, and your loss (if you are not lucky, and the buffers don't overlap) will be significant. — Alex Cohn, Feb 04 '20 at 18:48
*One more comment:* I would expect the result of overlap check to be consistent for a given device, and even same for same devices (e.g. if one Pixel 3 with Andorid 10 doesn't show overlap, you won't see the buffers overlap on another Pixel 3 with Android 10). — Alex Cohn, Feb 04 '20 at 18:51
yBufferPos needs to advance by `rowStride` each iteration (like answer below). In this code it advances by `rowStride-width`, I think it is incorrect — Sarsaparilla, Aug 16 '20 at 05:09
@Sarsaparilla I bet you are right. Got lost in refactoring. Fixed. Thanks for attention to detail! — Alex Cohn, Aug 16 '20 at 07:13

score 3 · Answer 2 · answered May 19 '20 at 10:26

Based on @Alex Cohn answer, I have implemented it in the JNI part, trying to take profit from the byte-access and performance advantages. I left it here, maybe it could be as useful as the @Alex answer was for me. It's almost the same algorithm, in C; based in an image with YUV_420_888 format:

uchar* yuvToNV21(jbyteArray yBuf, jbyteArray uBuf, jbyteArray vBuf, jbyte *fullArrayNV21,
    int width, int height, int yRowStride, int yPixelStride, int uRowStride,
    int uPixelStride, int vRowStride, int vPixelStride, JNIEnv *env) {

    /* Check that our frame has right format, as specified at android docs for
     * YUV_420_888 (https://developer.android.com/reference/android/graphics/ImageFormat?authuser=2#YUV_420_888):
     *      - Plane Y not overlaped with UV, and always with pixelStride = 1
     *      - Planes U and V have the same rowStride and pixelStride (overlaped or not)
     */
    if(yPixelStride != 1 || uPixelStride != vPixelStride || uRowStride != vRowStride) {
        jclass Exception = env->FindClass("java/lang/Exception");
        env->ThrowNew(Exception, "Invalid YUV_420_888 byte structure. Not agree with https://developer.android.com/reference/android/graphics/ImageFormat?authuser=2#YUV_420_888");
    }

    int ySize = width*height;
    int uSize = env->GetArrayLength(uBuf);
    int vSize = env->GetArrayLength(vBuf);
    int newArrayPosition = 0; //Posicion por la que vamos rellenando el array NV21
    if (fullArrayNV21 == nullptr) {
        fullArrayNV21 = new jbyte[ySize + uSize + vSize];
    }
    if(yRowStride == width) {
        //Best case. No padding, copy direct
        env->GetByteArrayRegion(yBuf, newArrayPosition, ySize, fullArrayNV21);
        newArrayPosition = ySize;
    }else {
        // Padding at plane Y. Copy Row by Row
        long yPlanePosition = 0;
        for(; newArrayPosition<ySize; newArrayPosition += width) {
            env->GetByteArrayRegion(yBuf, yPlanePosition, width, fullArrayNV21 + newArrayPosition);
            yPlanePosition += yRowStride;
        }
    }

    // Check UV channels in order to know if they are overlapped (best case)
    // If they are overlapped, U and B first bytes are consecutives and pixelStride = 2
    long uMemoryAdd = (long)&uBuf;
    long vMemoryAdd = (long)&vBuf;
    long diff = std::abs(uMemoryAdd - vMemoryAdd);
    if(vPixelStride == 2 && diff == 8) {
        if(width == vRowStride) {
            // Best Case: Valid NV21 representation (UV overlapped, no padding). Copy direct
            env->GetByteArrayRegion(uBuf, 0, uSize, fullArrayNV21 + ySize);
            env->GetByteArrayRegion(vBuf, 0, vSize, fullArrayNV21 + ySize + uSize);
        }else {
            // UV overlapped, but with padding. Copy row by row (too much performance improvement compared with copy byte-by-byte)
            int limit = height/2 - 1;
            for(int row = 0; row<limit; row++) {
                env->GetByteArrayRegion(uBuf, row * vRowStride, width, fullArrayNV21 + ySize + (row * width));
            }
        }
    }else {
        //WORST: not overlapped UV. Copy byte by byte
        for(int row = 0; row<height/2; row++) {
           for(int col = 0; col<width/2; col++) {
               int vuPos = col*uPixelStride + row*uRowStride;
               env->GetByteArrayRegion(vBuf, vuPos, 1, fullArrayNV21 + newArrayPosition);
               newArrayPosition++;
               env->GetByteArrayRegion(uBuf, vuPos, 1, fullArrayNV21 + newArrayPosition);
               newArrayPosition++;
           }
        }
    }
    return (uchar*)fullArrayNV21;
}

I'm sure that some improvements can be added, but I have tested in a lot of devices, and it is working with very good performance and stability.

Thank you for the snippet. It works great. I had to make some modifications though. Because of compiler optimizations, the calculated diff from uMemoryAdd and yMemoryAdd may be different between optimized and not optimized code. I replaced it with checking uBuf[0] == vBuf[1] instead. The other thing i changed is remove the -1 in int limit = height/2 - 1; because the for loop was not processing the last row, leaving a green line at the bottom as a result. — Martyns, May 28 '20 at 21:33
@Juan Miguel S. I tested your code with camera preview size 176*144 pixel, it has the green line at the bottom and right of image. Change like Martyns cause crash. — DzungPV, May 25 '22 at 12:17

score 1 · Answer 3 · answered Dec 13 '19 at 10:48

    public static byte[] YUV420toNV21(Image image) {
        Rect crop = image.getCropRect();
        int format = image.getFormat();
        int width = crop.width();
        int height = crop.height();
        Image.Plane[] planes = image.getPlanes();
        byte[] data = new byte[width * height * ImageFormat.getBitsPerPixel(format) / 8];
        byte[] rowData = new byte[planes[0].getRowStride()];

        int channelOffset = 0;
        int outputStride = 1;
        for (int i = 0; i < planes.length; i++) {
            switch (i) {
                case 0:
                    channelOffset = 0;
                    outputStride = 1;
                    break;
                case 1:
                    channelOffset = width * height + 1;
                    outputStride = 2;
                    break;
                case 2:
                    channelOffset = width * height;
                    outputStride = 2;
                    break;
            }

            ByteBuffer buffer = planes[i].getBuffer();
            int rowStride = planes[i].getRowStride();
            int pixelStride = planes[i].getPixelStride();

            int shift = (i == 0) ? 0 : 1;
            int w = width >> shift;
            int h = height >> shift;
            buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
            for (int row = 0; row < h; row++) {
                int length;
                if (pixelStride == 1 && outputStride == 1) {
                    length = w;
                    buffer.get(data, channelOffset, length);
                    channelOffset += length;
                } else {
                    length = (w - 1) * pixelStride + 1;
                    buffer.get(rowData, 0, length);
                    for (int col = 0; col < w; col++) {
                        data[channelOffset] = rowData[col * pixelStride];
                        channelOffset += outputStride;
                    }
                }
                if (row < h - 1) {
                    buffer.position(buffer.position() + rowStride - length);
                }
            }
        }
        return data;
    }

How does this piece of code solve the problem? Please elaborate. — Orestis Zekai, Dec 13 '19 at 10:52

Finley · Answer 4 · 2022-12-03T13:15:36.990

0

Check this answer for an acceleration method when it is not disguised NV21 or rowStride != width(there are paddings on each line).

edited Dec 03 '22 at 13:15

answered Dec 03 '22 at 08:25

Finley

795
1
8
26

camera2 captured picture - conversion from YUV_420_888 to NV21

4 Answers4

Linked