AVAssetReader trouble getting pixel buffer from copyNextSampleBuffer(), Swift

Question

I'm trying to read the image frames from a Quicktime movie file using AVFoundation and AVAssetReader on macOSX. I want to display the frames via a texture map in Metal. There are many examples of using AVAssetReader online, but I cannot get it working for what I want.

I can read the basic frame data from the movie -- the time values, size, and durations in the printout look correct. However, when I try to get the pixelBuffer, CMSampleBufferGetImageBuffer returns NULL.

    let track = asset.tracks(withMediaType: AVMediaType.video)[0]
    let videoReaderSettings : [String : Int] = [kCVPixelBufferPixelFormatTypeKey as String : Int(kCVPixelFormatType_32BGRA)]

    let output = AVAssetReaderTrackOutput(track:track, outputSettings:nil)  // using videoReaderSettings causes it to no longer report frame data
    guard let reader = try? AVAssetReader(asset: asset) else {exit(1)}
    output.alwaysCopiesSampleData = true
    reader.add(output)
    reader.startReading()


    while(reader.status == .reading){

        if let sampleBuffer = output.copyNextSampleBuffer(), CMSampleBufferIsValid(sampleBuffer) {
            let frameTime = CMSampleBufferGetOutputPresentationTimeStamp(sampleBuffer)
            if (frameTime.isValid){
                print("frame: \(frameNumber), time: \(String(format:"%.3f", frameTime.seconds)), size: \(CMSampleBufferGetTotalSampleSize(sampleBuffer)), duration: \(                CMSampleBufferGetOutputDuration(sampleBuffer).value)")

                if let pixelBuffer : CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
                    getTextureFromCVBuffer(pixelBuffer)
                    //   break

                }
                frameNumber += 1
            }
        }
    }

This problem was addressed here (Why does CMSampleBufferGetImageBuffer return NULL) where it is suggested that the problem is that one must specify a video format in the settings argument instead of 'nil'. So I tried replacing 'nil' with 'videoReaderSettings' above, with various values for the format: kCVPixelFormatType_32BGRA, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, and others.

The result is that the frame 'time' values are still correct, but the 'size' and 'duration' values are 0. However, CMSampleBufferGetImageBuffer DOES return something, where before it was 0. But garbage shows up onscreen.

Here is the function which converts the pixelBuffer to a Metal texture.

func getTextureFromCVBuffer(_ pixelBuffer:CVPixelBuffer) {
    // Get width and height for the pixel buffer
    let width = CVPixelBufferGetWidth(pixelBuffer)
    let height = CVPixelBufferGetHeight(pixelBuffer)

    // Converts the pixel buffer in a Metal texture.
    var cvTextureOut: CVMetalTexture?
    if CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, self.textureCache!, pixelBuffer, nil, .bgra8Unorm, width, height, 0, &cvTextureOut) != kCVReturnSuccess {
        print ("CVMetalTexture create failed!")
    }

    guard let cvTexture = cvTextureOut, let inputTexture = CVMetalTextureGetTexture(cvTexture) else {
        print("Failed to create metal texture")
        return
    }
    texture = inputTexture
}

When I'm able to pass a pixelBuffer to this function, it does report the correct size for the image. But as I said, what appears onscreen is garbage -- its composed of chunks of recent Safari browser pages actually. I'm not sure if the problem is in the first function or the second function. A nonzero return value from CMSampleBufferGetImageBuffer is encouraging, but the 0's for size and duration are not.

I found this thread (Buffer size of CMSampleBufferRef) which suggests that showing 0 for the size and duration may not be a problem, so maybe the issue is in the conversion to the Metal texture?

Any idea what I am doing wrong?
Thanks!

score 0 · Answer 1 · answered Jul 03 '20 at 04:51

put videoReaderSetting at AVAssetReaderTrackOutput.

let videoReaderSettings : [String : Int] = [kCVPixelBufferPixelFormatTypeKey as String : Int(kCVPixelFormatType_32BGRA)]

let output = AVAssetReaderTrackOutput(track:track, outputSettings: videoReaderSettings)

AVAssetReader trouble getting pixel buffer from copyNextSampleBuffer(), Swift

1 Answers1