9

I've implemented previous suggestions with Swift (How to use CVPixelBufferPool in conjunction with AVAssetWriterInputPixelBufferAdaptor in iPhone?), but got stuck with an "kCVReturnInvalidArgument" (error value: -6661) when using CVPixelBufferPoolCreatePixelBuffer as guided.

I'm basically trying to create a movie from images, but as the buffer pool isn't created successfully, I can't append pixel buffers--here is my code for doing this.

Any suggestions are highly appreciated!

import Foundation
import Photos
import OpenGLES
import AVFoundation
import CoreMedia

class MovieGenerator {

    var _videoWriter:AVAssetWriter
    var _videoWriterInput: AVAssetWriterInput
    var _adapter: AVAssetWriterInputPixelBufferAdaptor
    var _buffer = UnsafeMutablePointer<Unmanaged<CVPixelBuffer>?>.alloc(1)


    init(frameSize size: CGSize, outputURL url: NSURL) {

    // delete file if exists
    let sharedManager = NSFileManager.defaultManager() as NSFileManager
    if(sharedManager.fileExistsAtPath(url.path!)) {
        sharedManager.removeItemAtPath(url.path, error: nil)
    }

    // video writer
    _videoWriter = AVAssetWriter(URL: url, fileType: AVFileTypeQuickTimeMovie, error: nil)

    // writer input
    var videoSettings = [AVVideoCodecKey:AVVideoCodecH264, AVVideoWidthKey:size.width, AVVideoHeightKey:size.height]
    _videoWriterInput = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: videoSettings)
    _videoWriterInput.expectsMediaDataInRealTime = true
    _videoWriter.addInput(_videoWriterInput)

    // pixel buffer adapter
    var adapterAttributes = [kCVPixelBufferPixelFormatTypeKey:kCVPixelFormatType_32BGRA, kCVPixelBufferWidthKey: size.width,
        kCVPixelBufferHeightKey: size.height,
        kCVPixelFormatOpenGLESCompatibility: kCFBooleanTrue]

    _adapter = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: _videoWriterInput, sourcePixelBufferAttributes: adapterAttributes)
    var poolCreateResult:CVReturn = CVPixelBufferPoolCreatePixelBuffer(nil, _adapter.pixelBufferPool, _buffer)
    println("pool creation:\(poolCreateResult)")

    _videoWriter.startWriting()
    _videoWriter.startSessionAtSourceTime(kCMTimeZero)

}

func addImage(image:UIImage, frameNum:Int, fps:Int)->Bool {


    self.createPixelBufferFromCGImage(image.CGImage, pixelBufferPtr: _buffer)

    var presentTime:CMTime = CMTimeMake(Int64(frameNum), Int32(fps))
    var result:Bool = _adapter.appendPixelBuffer(_buffer.memory?.takeUnretainedValue(), withPresentationTime: presentTime)

    return result
}

func finalizeMovie(timeStamp: CMTime) {
    _videoWriterInput.markAsFinished()
    _videoWriter.endSessionAtSourceTime(timeStamp)
    _videoWriter.finishWritingWithCompletionHandler({println("video writer finished with status: \(self._videoWriter.status)")})
}

func createPixelBufferFromCGImage(image: CGImage, pixelBufferPtr: UnsafeMutablePointer<Unmanaged<CVPixelBuffer>?>) {

    let width:UInt = CGImageGetWidth(image)
    let height:UInt = CGImageGetHeight(image)

    let imageData:CFData = CGDataProviderCopyData(CGImageGetDataProvider(image))
    let options:CFDictionary = [kCVPixelBufferCGImageCompatibilityKey:NSNumber.numberWithBool(true), kCVPixelBufferCGBitmapContextCompatibilityKey:NSNumber.numberWithBool(true)]

    var status:CVReturn = CVPixelBufferCreate(kCFAllocatorDefault, width, height, OSType(kCVPixelFormatType_32BGRA), options, pixelBufferPtr)
    assert(status != 0,"CVPixelBufferCreate: \(status)")

    var lockStatus:CVReturn = CVPixelBufferLockBaseAddress(pixelBufferPtr.memory?.takeUnretainedValue(), 0)
    println("CVPixelBufferLockBaseAddress: \(lockStatus)")

    var pxData:UnsafeMutablePointer<(Void)> = CVPixelBufferGetBaseAddress(pixelBufferPtr.memory?.takeUnretainedValue())
    let bitmapinfo = CGBitmapInfo.fromRaw(CGImageAlphaInfo.NoneSkipFirst.toRaw())
    let rgbColorSpace:CGColorSpace = CGColorSpaceCreateDeviceRGB()

    var context:CGContextRef = CGBitmapContextCreate(pxData, width, height, 8, 4*CGImageGetWidth(image), rgbColorSpace, bitmapinfo!)

    CGContextDrawImage(context, CGRectMake(0, 0, CGFloat(width), CGFloat(height)), image)

    CVPixelBufferUnlockBaseAddress(pixelBufferPtr.memory?.takeUnretainedValue(), 0)


}



}
Yoon Bai
  • 91
  • 1
  • 3
  • I solved (or, more accurately, worked around) the problem by implementing my 'createPixelBufferFromCGImage` function in obj-c and calling `CVPixelBufferPoolCreatePixelBuffer` from there. My guess is that our Swift `CVPixelBuffer` pointers are somehow incompatible with `CVPixelBufferPoolCreatePixelBuffer`, thus creating the "invalid argument" error. If I get a pure Swift version working, I'll post an answer. – acj May 11 '15 at 11:24

2 Answers2

5

I can't exactly answer your question, frustratingly, but I am working on code that does essentially the same thing. And, mine happens to get further than the error you have been getting; it gets all the way to the point where it's attempting to add the images to the movie and then simply fails by never getting a successful result from appendPixelBuffer() -- and I'm not sure how to figure out why. I'm posting this in the hopes that it helps you get further, though.

(My code is adapted from AVFoundation + AssetWriter: Generate Movie With Images and Audio, and I used your post to help navigate som e of the pointer interop shenanigans...)

func writeAnimationToMovie(path: String, size: CGSize, animation: Animation) -> Bool {
    var error: NSError?
    let writer = AVAssetWriter(URL: NSURL(fileURLWithPath: path), fileType: AVFileTypeQuickTimeMovie, error: &error)

    let videoSettings = [AVVideoCodecKey: AVVideoCodecH264, AVVideoWidthKey: size.width, AVVideoHeightKey: size.height]

    let input = AVAssetWriterInput(mediaType: AVMediaTypeVideo, outputSettings: videoSettings)
    let pixelBufferAdaptor = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: input, sourcePixelBufferAttributes: nil)
    input.expectsMediaDataInRealTime = true
    writer.addInput(input)

    writer.startWriting()
    writer.startSessionAtSourceTime(kCMTimeZero)

    var buffer: CVPixelBufferRef

    var frameCount = 0
    for frame in animation.frames {
        let rect = CGRectMake(0, 0, size.width, size.height)
        let rectPtr = UnsafeMutablePointer<CGRect>.alloc(1)
        rectPtr.memory = rect
        buffer = pixelBufferFromCGImage(frame.image.CGImageForProposedRect(rectPtr, context: nil, hints: nil).takeUnretainedValue(), size)
        var appendOk = false
        var j = 0
        while (!appendOk && j < 30) {
            if pixelBufferAdaptor.assetWriterInput.readyForMoreMediaData {
                let frameTime = CMTimeMake(Int64(frameCount), 10)
                appendOk = pixelBufferAdaptor.appendPixelBuffer(buffer, withPresentationTime: frameTime)
                // appendOk will always be false
                NSThread.sleepForTimeInterval(0.05)
            } else {
                NSThread.sleepForTimeInterval(0.1)
            }
            j++
        }
        if (!appendOk) {
            println("Doh, frame \(frame) at offset \(frameCount) failed to append")
        }
    }

    input.markAsFinished()
    writer.finishWritingWithCompletionHandler({
        if writer.status == AVAssetWriterStatus.Failed {
            println("oh noes, an error: \(writer.error.description)")
        } else {
            println("hrmmm, there should be a movie?")
        }
    })

    return true;
}

Where pixelBufferFromCGImage is defined like so:

func pixelBufferFromCGImage(image: CGImageRef, size: CGSize) -> CVPixelBufferRef {
    let options = [
        kCVPixelBufferCGImageCompatibilityKey: true,
        kCVPixelBufferCGBitmapContextCompatibilityKey: true]
    var pixBufferPointer = UnsafeMutablePointer<Unmanaged<CVPixelBuffer>?>.alloc(1)

    let status = CVPixelBufferCreate(
        nil,
        UInt(size.width), UInt(size.height),
        OSType(kCVPixelFormatType_32ARGB),
        options,
        pixBufferPointer)

    CVPixelBufferLockBaseAddress(pixBufferPointer.memory?.takeUnretainedValue(), 0)

    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let bitmapinfo = CGBitmapInfo.fromRaw(CGImageAlphaInfo.NoneSkipFirst.toRaw())

    var pixBufferData:UnsafeMutablePointer<(Void)> = CVPixelBufferGetBaseAddress(pixBufferPointer.memory?.takeUnretainedValue())

    let context = CGBitmapContextCreate(
        pixBufferData,
        UInt(size.width), UInt(size.height),
        8, UInt(4 * size.width),
        rgbColorSpace, bitmapinfo!)

    CGContextConcatCTM(context, CGAffineTransformMakeRotation(0))
    CGContextDrawImage(
        context,
        CGRectMake(0, 0, CGFloat(CGImageGetWidth(image)), CGFloat(CGImageGetHeight(image))),
        image)

    CVPixelBufferUnlockBaseAddress(pixBufferPointer.memory?.takeUnretainedValue(), 0)
    return pixBufferPointer.memory!.takeUnretainedValue()
}
Community
  • 1
  • 1
Eric O'Connell
  • 858
  • 5
  • 14
  • 1
    Ah, silly me. I hadn't actually called `markAsFinished` on the input and `finishWritingWithCompletionHandler()` on the writer! – Eric O'Connell Sep 09 '14 at 07:05
  • Did you ever get this to work? I'm working on the same thing (porting working Objective C code to Swift). –  Jan 30 '15 at 02:07
5

Per the docs for pixelBufferPool:

This property is NULL before the first call to startSessionAtTime:on the associated AVAssetWriter object.

Moving the call to CVPixelBufferPoolCreatePixelBuffer to the end of init should fix the immediate problem.

A few other observations:

  • You have your AVAssetWriterInputPixelBufferAdaptor configured for BGRA, but in createPixelBufferFromCGImage you're using RGB. Your final videos will look strange if the pixel formats are mismatched.
  • You don't need to call CVPixelBufferCreate in your createPixelBufferFromCGImage method. This defeats the purpose of using the buffer pool.
  • If you're running this in a tight loop, memory consumption will become a problem. Using autoreleasepool and being careful with takeUnretainedValue vs takeRetainedValue will help.

I've posted reference implementations for Swift 1.2, 2.0, and 3.0 that use buffer pools.

acj
  • 4,821
  • 4
  • 34
  • 49
  • what if you set the attributes for AVAssetWriterInputPixelBufferAdaptor to nil instead? other SO answers seem to do that. mind explaining why you explicitly set the attributes? thanks for sharing code! – Crashalot Mar 29 '16 at 00:56
  • also in your 2.0 code why do you set frameCount = Int64(0) inside the block of requestMediaDataWhenReadyOnQueue? Doesn't this mean each time the block gets invoked that frameCount gets reset to 0? For instance, if you write 100 images out and then readyForMoreMediaData turns false, when you start again on image 101, frameCount will reset to 0 instead of being 101? – Crashalot Mar 29 '16 at 01:39
  • @Crashalot You can probably use `nil` for the attributes. I had done some experimenting with color spaces and different dimensions and just left the code in place. // For the photo sets I used in my testing, `readyForMoreMediaData` was always true and didn't trigger the scenario that you're describing. You're right that it could present a problem, though, and for the sake of robustness you probably should maintain that state (`frameCount`, etc) outside of the block. – acj Mar 29 '16 at 21:22
  • OK cool, that was the assumption. Then the same should hold true for marking the writer as finished? Would you like a modified version of the code that accounts for these changes? Thanks so much for sharing. It was awesome! Also using `nil` produced a video with only a green screen so it appears you must define the AVAssetWriterInputPixelBufferAdaptor attributes, at least for your code. – Crashalot Mar 29 '16 at 21:51
  • finished porting a Swift version, please suggest improvements if you see problems: http://stackoverflow.com/questions/3741323/how-do-i-export-uiimage-array-as-a-movie/36297656#36297656 – Crashalot Mar 30 '16 at 00:44