CMSampleBuffer frame converted to vImage has wrong colors

Question

I’m trying to convert CMSampleBuffer from camera output to vImage and later apply some processing. Unfortunately, even without any further editing, frame I get from buffer has wrong colors:

Implementation (Memory management and errors are not considered in question):

Configuring video output device:

    videoDataOutput = AVCaptureVideoDataOutput()
    videoDataOutput.videoSettings = [String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA]
    videoDataOutput.alwaysDiscardsLateVideoFrames = true
    videoDataOutput.setSampleBufferDelegate(self, queue: captureQueue)
    videoConnection = videoDataOutput.connection(withMediaType:  AVMediaTypeVideo)

    captureSession.sessionPreset = AVCaptureSessionPreset1280x720

    let videoDevice = AVCaptureDevice.defaultDevice(withMediaType:  AVMediaTypeVideo)
    guard let videoDeviceInput = try? AVCaptureDeviceInput(device: videoDevice) else {
        return
    }

Creating vImage from CASampleBuffer received from camera:

   // Convert `CASampleBuffer` to `CVImageBuffer`
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

    var buffer: vImage_Buffer = vImage_Buffer()
    buffer.data = CVPixelBufferGetBaseAddress(pixelBuffer)
    buffer.rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer)
    buffer.width = vImagePixelCount(CVPixelBufferGetWidth(pixelBuffer))
    buffer.height = vImagePixelCount(CVPixelBufferGetHeight(pixelBuffer))

    let vformat = vImageCVImageFormat_CreateWithCVPixelBuffer(pixelBuffer)
    let bitmapInfo:CGBitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)

    var cgFormat = vImage_CGImageFormat(bitsPerComponent: 8,
                                        bitsPerPixel: 32,
                                        colorSpace: nil,
                                        bitmapInfo: bitmapInfo,
                                        version: 0,
                                        decode: nil,
                                        renderingIntent: .defaultIntent)

     // Create vImage
     vImageBuffer_InitWithCVPixelBuffer(&buffer, &cgFormat, pixelBuffer, vformat!.takeRetainedValue(), cgColor, vImage_Flags(kvImageNoFlags))

Converting buffer to UIImage:

For the sake of tests CVPixelBuffer is exported to UIImage, but adding it to video buffer has the same result.

var dstPixelBuffer: CVPixelBuffer?

    let status = CVPixelBufferCreateWithBytes(nil, Int(buffer.width), Int(buffer.height),
                                              kCVPixelFormatType_32BGRA, buffer.data,
                                              Int(buffer.rowBytes), releaseCallback,
                                              nil, nil, &dstPixelBuffer)

    let destCGImage = vImageCreateCGImageFromBuffer(&buffer, &cgFormat, nil, nil, numericCast(kvImageNoFlags), nil)?.takeRetainedValue()

    // create a UIImage
    let exportedImage = destCGImage.flatMap { UIImage(cgImage: $0, scale: 0.0, orientation: UIImageOrientation.right) }

    DispatchQueue.main.async {
        self.previewView.image = exportedImage
    }

Does removing `| CGBitmapInfo.byteOrder32Little.rawValue` help? Your yellow duck went from FFFF00 to 00FF00, so that _could_ be a byte swapping problem, if you include a zero alpha. — Rhythmic Fistman, Sep 10 '17 at 14:09
@RhythmicFistman that's interesting. I tried to remove `byteOrder32Little` and also change to `CGBitmapInfo(rawValue: CGImageByteOrderInfo.orderMask.rawValue | CGImageAlphaInfo.last.rawValue)` to keep default order, but result seems exactly the same in all cases. — Dawid, Sep 10 '17 at 18:45

Flex Monkey · Accepted Answer · 2018-01-11T17:59:58.153

Try setting the color space on your CV image format:

    let vformat = vImageCVImageFormat_CreateWithCVPixelBuffer(pixelBuffer).takeRetainedValue()

    vImageCVImageFormat_SetColorSpace(vformat,
                                      CGColorSpaceCreateDeviceRGB())

...and update your call to vImageBuffer_InitWithCVPixelBuffer to reflect the fact vformat is now a managed reference:

    let error = vImageBuffer_InitWithCVPixelBuffer(&buffer, &cgFormat, pixelBuffer, vformat, nil, vImage_Flags(kvImageNoFlags))

Finally, your can remove the following lines, vImageBuffer_InitWithCVPixelBuffer is doing that work for you:

//        buffer.data = CVPixelBufferGetBaseAddress(pixelBuffer)
//        buffer.rowBytes = CVPixelBufferGetBytesPerRow(pixelBuffer)
//        buffer.width = vImagePixelCount(CVPixelBufferGetWidth(pixelBuffer))
//        buffer.height = vImagePixelCount(CVPixelBufferGetHeight(pixelBuffer))

Note that you don't need to lock the Core Video pixel buffer, if you check the headerdoc, it says "It is not necessary to lock the CVPixelBuffer before calling this function".

I came back to this problem and just found your advice. Setting color space that was a missing puzzle! Thank you very much for all tips!!! — Dawid, May 24 '18 at 09:30

score 0 · Answer 2 · answered Sep 11 '17 at 12:44

The call to vImageBuffer_InitWithCVPixelBuffer is performing modifying your vImage_Buffer and CVPixelBuffer's contents, which is a bit naughty because in your (linked) code you promise not to modify the pixels when you say

CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)

The correct way to initialise the CGBitmapInfo for BGRA8888 is alpha first, 32bit little endian, which is non obvious, but covered in the header file for vImage_CGImageFormat in vImage_Utilities.h:

let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.first.rawValue | CGImageByteOrderInfo.order32Little.rawValue)

What I don't get is why vImageBuffer_InitWithCVPixelBuffer is modifying your buffer, as cgFormat (desiredFormat) should match vformat, although it is documented to modify the buffer, so maybe you should copy the data first.

CMSampleBuffer frame converted to vImage has wrong colors

2 Answers2