iOS using vImage - Accelerate to convert QCAR YUV to RGB

Question

I'm trying to test the performance of converting YUV images produced by Vuforia and converting them to UIImage using the iOS Accelerate Framework's vImage calls. In the current state of the code I'm just trying to get it to work. Right now converting yields a dark striped image. Are there any published details about how Vuforia has laid out the YUV format in their implementation? My initial assumption was that they used the bi-planar 420p format iOS devices use. Relevant test code follows.

UIImage *imageWithQCARCameraImage(const QCAR::Image *cameraImage)
{
    UIImage *image = nil;

    if (cameraImage) {
        QCAR::PIXEL_FORMAT pixelFormat = cameraImage->getFormat();

        CGColorSpaceRef colorSpace = NULL;
        switch (pixelFormat) {
            case QCAR::YUV:
            case QCAR::RGB888:
                colorSpace = CGColorSpaceCreateDeviceRGB();
                break;
            case QCAR::GRAYSCALE:
                colorSpace = CGColorSpaceCreateDeviceGray();
                break;
            case QCAR::RGB565:
            case QCAR::RGBA8888:
            case QCAR::INDEXED:
                std::cerr << "Image format conversion not implemented." << std::endl;
                break;
            case QCAR::UNKNOWN_FORMAT:
                std::cerr << "Image format unknown." << std::endl;
                break;
        }

        int bitsPerComponent = 8;
        int width = cameraImage->getWidth();
        int height = cameraImage->getHeight();
        const void *baseAddress = cameraImage->getPixels();
        size_t totalBytes = QCAR::getBufferSize(width, height, pixelFormat);

        CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault | kCGImageAlphaNone;
        CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
        CGImageRef imageRef = NULL;

        if (pixelFormat == QCAR::YUV) {
            int bytesPerPixel = 4;
            uint8_t *sourceDataAddress = (uint8_t *)baseAddress;

            static vImage_Buffer srcYp = {
                .width = static_cast<vImagePixelCount>(width),
                .height = static_cast<vImagePixelCount>(height),
                .data = const_cast<void *>(baseAddress)
            };

            size_t lumaBytes = width * height;
            size_t chromianceBytes = totalBytes - lumaBytes;
            static vImage_Buffer srcCb = {
                .data = static_cast<void *>(sourceDataAddress + lumaBytes)
            };

            static vImage_Buffer srcCr = {
                .data = static_cast<void *>(sourceDataAddress + lumaBytes + (chromianceBytes / 2))
            };

            static vImage_Buffer dest = {
                .width = static_cast<vImagePixelCount>(width),
                .height = static_cast<vImagePixelCount>(height),
                .data = imageData
            };

            //uint8_t permuteMap[] = { 1, 2, 3, 0 };
            vImage_YpCbCrPixelRange pixelRange = (vImage_YpCbCrPixelRange){ 0, 128, 255, 255, 255, 1, 255, 0 };
            vImage_YpCbCrToARGB info;

            vImage_Error error;

            error = vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4,
                                                                  &pixelRange,
                                                                  &info,
                                                                  kvImage420Yp8_Cb8_Cr8,
                                                                  kvImageARGB8888,
                                                                  kvImagePrintDiagnosticsToConsole);


            error = vImageConvert_420Yp8_Cb8_Cr8ToARGB8888(&srcYp,
                                                           &srcCb,
                                                           &srcCr,
                                                           &dest,
                                                           &info,
                                                           NULL,
                                                           1,
                                                           kvImageNoFlags);

            vImage_CGImageFormat format =
            {
                .bitsPerComponent = static_cast<uint32_t>(bitsPerComponent),
                .bitsPerPixel = static_cast<uint32_t>(3 * bitsPerComponent),
                .colorSpace = colorSpace,
                .bitmapInfo = bitmapInfo,
                .version = 0,
                .decode = NULL,
                .renderingIntent = renderingIntent
            };

            imageRef = vImageCreateCGImageFromBuffer(&dest,
                                                     &format,
                                                     NULL,
                                                     NULL,
                                                     kvImageNoFlags,
                                                     &error);
            if (error) {
                std::cerr << "Err." << std::endl;
            }
        } else {
            int bitsPerPixel = QCAR::getBitsPerPixel(pixelFormat);
            int bytesPerRow = cameraImage->getStride();
            CGDataProviderRef provider = CGDataProviderCreateWithData(NULL,
                                                                      baseAddress,
                                                                      totalBytes,
                                                                      NULL);

           imageRef = CGImageCreate(width,
                                    height,
                                    bitsPerComponent,
                                    bitsPerPixel,
                                    bytesPerRow,
                                    colorSpace,
                                    bitmapInfo,
                                    provider,
                                    NULL,
                                    false,
                                    renderingIntent);
            CGDataProviderRelease(provider);
        }

        if (imageRef != NULL) {
            image = [UIImage imageWithCGImage:imageRef];
            CGImageRelease(imageRef);
        }

        if (colorSpace != NULL) {
            CGColorSpaceRelease(colorSpace);
        }

    }

    return image;
}

Any published details from Qualcomm has so far been nothing but useless. But in their forums, the moderators "try" to answer questions. If it's a good question there's only two ways they will respond, "Refer to documentation" or an actual answer. Hope you get your answer though, sounds interesting. — Augmented Jacob, Apr 08 '15 at 09:14
I'll ask on the forum too, although their forum isn't great. What I have learned about YUV tells me that their API doesn't provide enough detail for you to do anything but guess since there are a myriad of potential formats. — Cameron Lowell Palmer, Apr 08 '15 at 09:20
You need to fill out all four fields of the vImage_Buffer struct every time. — Ian Ollmann, Apr 17 '15 at 15:24
@IanOllmann that is why I stated just for testing. If however, you're going to make such a claim for functionality it would be helpful if you would provide a citation, if you're talking about performance then that is why I stated it was for testing. Make it work, make it right, make it fast. In that order. — Cameron Lowell Palmer, Apr 19 '15 at 08:23
I am my own citation on this. Burning bushes will be supplied as required. — Ian Ollmann, Apr 20 '15 at 18:11
Ian is the canonical authority on the subject of all things vImage. When he demands sacrifice, you find a goat. — Stephen Canon, Apr 21 '15 at 04:40
However, doesn't answer the basic question to this which is... What is the format of a Vuforia YUV frame? — Cameron Lowell Palmer, Apr 21 '15 at 11:07

score 4 · Answer 1 · answered Dec 05 '18 at 09:21

void *baseAddress = buffer;
size_t totalBytes = width * height * 3 / 2;

uint8_t *sourceDataAddress = (uint8_t *)baseAddress;

vImage_Buffer srcYp = {
    .width = static_cast<vImagePixelCount>(width),
    .height = static_cast<vImagePixelCount>(height),
    .rowBytes = static_cast<size_t>(width),
    .data = const_cast<void *>(baseAddress),
};

size_t lumaBytes = width * height;
size_t chromianceBytes = totalBytes - lumaBytes;
vImage_Buffer srcCb = {
    .width = static_cast<vImagePixelCount>(width) / 2,
    .height = static_cast<vImagePixelCount>(height) / 2,
    .rowBytes = static_cast<size_t>(width) / 2,
    .data = static_cast<void *>(sourceDataAddress + lumaBytes),
};

vImage_Buffer srcCr = {
    .width = static_cast<vImagePixelCount>(width) / 2,
    .height = static_cast<vImagePixelCount>(height) / 2,
    .rowBytes = static_cast<size_t>(width) / 2,
    .data = static_cast<void *>(sourceDataAddress + lumaBytes + (chromianceBytes / 2)),
};

vImage_Buffer dest;
dest.data = NULL;
vImage_Error error = kvImageNoError;
error = vImageBuffer_Init(&dest, height, width, 32, kvImageNoFlags);
// vImage_YpCbCrPixelRange pixelRange = (vImage_YpCbCrPixelRange){ 0, 128, 255, 255, 255, 1, 255, 0 };
vImage_YpCbCrPixelRange pixelRange = { 16, 128, 235, 240, 255, 0, 255, 0 };
vImage_YpCbCrToARGB info;
error = kvImageNoError;
error = vImageConvert_YpCbCrToARGB_GenerateConversion(kvImage_YpCbCrToARGBMatrix_ITU_R_601_4,
                                                      &pixelRange,
                                                      &info,
                                                      kvImage420Yp8_Cb8_Cr8,
                                                      kvImageARGB8888,
                                                      kvImagePrintDiagnosticsToConsole);
error = kvImageNoError;
uint8_t permuteMap[4] = {3, 2, 1, 0}; // BGRA - iOS only support BGRA
error = vImageConvert_420Yp8_Cb8_Cr8ToARGB8888(&srcYp,
                                               &srcCb,
                                               &srcCr,
                                               &dest,
                                               &info,
                                               permuteMap, // for iOS must be no NULL, mac can be NULL iOS only support BGRA
                                               255,
                                               kvImageNoFlags);

[edit] your answer and add explanation how/why this code solves the issue (above code, not somewhere in comments). — barbsan, Dec 05 '18 at 10:49
essentially, the op's mistake was not to divide by 2 the width and height in srcCb and srcCr. "The two chroma planes (blue and red projections) are sub-sampled in both the horizontal and vertical dimensions by a factor of 2". See https://wiki.videolan.org/YUV — Gibezynu Nu, Feb 05 '22 at 08:46

iOS using vImage - Accelerate to convert QCAR YUV to RGB

1 Answers1