How do I call CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer?

Question

I'm trying to figure out how to call this AVFoundation function in Swift. I've spent a ton of time fiddling with declarations and syntax, and got this far. The compiler is mostly happy, but I'm left with one last quandary.

public func captureOutput(
    captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
    fromConnection connection: AVCaptureConnection!
) {
    let samplesInBuffer = CMSampleBufferGetNumSamples(sampleBuffer)
    var audioBufferList: AudioBufferList

    var buffer: Unmanaged<CMBlockBuffer>? = nil

    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        nil,
        &audioBufferList,
        UInt(sizeof(audioBufferList.dynamicType)),
        nil,
        nil,
        UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
        &buffer
    )

    // do stuff
}

The compiler complains for the 3rd and 4th arguments:

Address of variable 'audioBufferList' taken before it is initialized

and

Variable 'audioBufferList' used before being initialized

So what am I supposed to do here?

I'm working off of this StackOverflow answer but it's Objective-C. I'm trying to translate it into Swift, but run into this problem.

Or is there possibly a better approach? I need to read the data from the buffer, one sample at a time, so I'm basically trying to get an array of the samples that I can iterate over.

Answer 1

Disclaimer: I have just tried to translate the code from Reading audio samples via AVAssetReader to Swift, and verified that it compiles. I have not tested if it really works.

// Needs to be initialized somehow, even if we take only the address
var audioBufferList = AudioBufferList(mNumberBuffers: 1,
      mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))

var buffer: Unmanaged<CMBlockBuffer>? = nil

CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
    sampleBuffer,
    nil,
    &audioBufferList,
    UInt(sizeof(audioBufferList.dynamicType)),
    nil,
    nil,
    UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
    &buffer
)

// Ensure that the buffer is released automatically.
let buf = buffer!.takeRetainedValue() 

// Create UnsafeBufferPointer from the variable length array starting at audioBufferList.mBuffers
let audioBuffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers,
    count: Int(audioBufferList.mNumberBuffers))

for audioBuffer in audioBuffers {
    // Create UnsafeBufferPointer<Int16> from the buffer data pointer
    var samples = UnsafeMutableBufferPointer<Int16>(start: UnsafeMutablePointer(audioBuffer.mData),
        count: Int(audioBuffer.mDataByteSize)/sizeof(Int16))

    for sample in samples {
        // ....
    }
}

Answer 2

Swift3 solution:

func loopAmplitudes(audioFileUrl: URL) {

    let asset = AVAsset(url: audioFileUrl)

    let reader = try! AVAssetReader(asset: asset)

    let track = asset.tracks(withMediaType: AVMediaTypeAudio)[0]

    let settings = [
        AVFormatIDKey : kAudioFormatLinearPCM
    ]

    let readerOutput = AVAssetReaderTrackOutput(track: track, outputSettings: settings)
    reader.add(readerOutput)
    reader.startReading()

    while let buffer = readerOutput.copyNextSampleBuffer() {

        var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
        var blockBuffer: CMBlockBuffer?

        CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
            buffer,
            nil,
            &audioBufferList,
            MemoryLayout<AudioBufferList>.size,
            nil,
            nil,
            kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
            &blockBuffer
        );

        let buffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))

        for buffer in buffers {

            let samplesCount = Int(buffer.mDataByteSize) / MemoryLayout<Int16>.size
            let samplesPointer = audioBufferList.mBuffers.mData!.bindMemory(to: Int16.self, capacity: samplesCount)
            let samples = UnsafeMutableBufferPointer<Int16>(start: samplesPointer, count: samplesCount)

            for sample in samples {

                //do something with you sample (which is Int16 amplitude value)

            }
        }
    }
}

Answer 3

The answers posted here make assumptions about the size of the necessary AudioBufferList -- which may have allowed them to have work in their particular circumstance, but didn't work for me when receiving audio from a AVCaptureSession. (Apple's own sample code didn't work either.)

The documentation on CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer is not obvious, but it turns out you can ask the function it how big AudioListBuffer item should be first, and then call it a second time with an AudioBufferList allocated to the size it wants.

Below is a C++ example (sorry, don't know Swift) that shows a more general solution that worked for me.

// ask the function how big the audio buffer list should be for this
// sample buffer ref
size_t requiredABLSize = 0;
err = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer,
                      &requiredABLSize,
                      NULL,
                      NULL,
                      kCFAllocatorSystemDefault,
                      kCFAllocatorSystemDefault,
                      kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                      NULL);

// allocate an audio buffer list of the required size
AudioBufferList* audioBufferList = (AudioBufferList*) malloc(requiredABLSize);
// ensure that blockBuffer is NULL in case the function fails
CMBlockBufferRef blockBuffer = NULL;

// now let the function allocate fill in the ABL for you
err = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer,
                      NULL,
                      audioBufferList,
                      requiredABLSize,
                      kCFAllocatorSystemDefault,
                      kCFAllocatorSystemDefault,
                      kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment,
                      &blockBuffer);

// if we succeeded...
if (err == noErr) {
   // la la la... read your samples...
}

// release the allocated block buffer
if (blockBuffer != NULL) {
    CFRelease(blockBuffer);
    blockBuffer = NULL;
}

// release the allocated ABL
if (audioBufferList != NULL) {
    free(audioBufferList);
    audioBufferList = NULL;
}

I'll leave it up to the Swift experts to offer an implementation in that language.

Answer 4

Martin's answer works and does exactly what I asked in the question, however, after posting the question and spending more time with the problem (and before seeing Martin's answer), I came up with this:

public func captureOutput(
    captureOutput: AVCaptureOutput!,
    didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
    fromConnection connection: AVCaptureConnection!
) {
    let samplesInBuffer = CMSampleBufferGetNumSamples(sampleBuffer)
    self.currentZ = Double(samplesInBuffer)

    let buffer: CMBlockBufferRef = CMSampleBufferGetDataBuffer(sampleBuffer)

    var lengthAtOffset: size_t = 0
    var totalLength: size_t = 0
    var data: UnsafeMutablePointer<Int8> = nil

    if( CMBlockBufferGetDataPointer( buffer, 0, &lengthAtOffset, &totalLength, &data ) != noErr ) {
        println("some sort of error happened")
    } else {
        for i in stride(from: 0, to: totalLength, by: 2) {
            // do stuff
        }
    }
}

This is a slightly different approach, and probably still has room for improvement, but the main point here is that at least on an iPad Mini (and probably other devices), each time this method is called, we get 1,024 samples. But those samples come in an array of 2,048 Int8 values. Every other one is the left/right byte that needs to be combined into to make an Int16 to turn the 2,048 half-samples into 1,024 whole samples.

Answer 5

it works for me. try it:

let musicUrl: NSURL = mediaItemCollection.items[0].valueForProperty(MPMediaItemPropertyAssetURL) as! NSURL
let asset: AVURLAsset = AVURLAsset(URL: musicUrl, options: nil)
let assetOutput = AVAssetReaderTrackOutput(track: asset.tracks[0] as! AVAssetTrack, outputSettings: nil)

var error : NSError?

let assetReader: AVAssetReader = AVAssetReader(asset: asset, error: &error)

if error != nil {
    print("Error asset Reader: \(error?.localizedDescription)")
}

assetReader.addOutput(assetOutput)
assetReader.startReading()

let sampleBuffer: CMSampleBufferRef = assetOutput.copyNextSampleBuffer()

var audioBufferList = AudioBufferList(mNumberBuffers: 1, mBuffers: AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil))
var blockBuffer: Unmanaged<CMBlockBuffer>? = nil


CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
    sampleBuffer,
    nil,
    &audioBufferList,
    sizeof(audioBufferList.dynamicType), // instead of UInt(sizeof(audioBufferList.dynamicType))
    nil,
    nil,
    UInt32(kCMSampleBufferFlag_AudioBufferList_Assure16ByteAlignment),
    &blockBuffer
)

Answer 6

I do this (swift 4.2):

let n = CMSampleBufferGetNumSamples(audioBuffer)
let format = CMSampleBufferGetFormatDescription(audioBuffer)!
let asbd = CMAudioFormatDescriptionGetStreamBasicDescription(format)!.pointee

let nChannels = Int(asbd.mChannelsPerFrame) // probably 2
let bufferlistSize = AudioBufferList.sizeInBytes(maximumBuffers: nChannels)
let abl = AudioBufferList.allocate(maximumBuffers: nChannels)
for i in 0..<nChannels {
    abl[i] = AudioBuffer(mNumberChannels: 0, mDataByteSize: 0, mData: nil)
}

var block: CMBlockBuffer?
var status = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(audioBuffer, bufferListSizeNeededOut: nil, bufferListOut: abl.unsafeMutablePointer, bufferListSize: bufferlistSize, blockBufferAllocator: nil, blockBufferMemoryAllocator: nil, flags: 0, blockBufferOut: &block)
assert(noErr == status)

// use AudioBufferList here (abl.unsafePointer), e.g. with ExtAudioFileWrite or what have you