Decode AAC to PCM format using AVAudioConverter Swift

Question

How convert AAC to PCM using AVAudioConverter, AVAudioCompressedBuffer and AVAudioPCMBuffer on Swift?

On WWDC 2015, 507 Session was said, that AVAudioConverter can encode and decode PCM buffer, was showed encode example, but wasn't showed examples with decoding. I tried decode, and something doesn't work. I don't know what:(

Calls:

//buffer - it's AVAudioPCMBuffer from AVAudioInputNode(AVAudioEngine)
let aacBuffer = AudioBufferConverter.convertToAAC(from: buffer, error: nil) //has data
let data = Data(bytes: aacBuffer!.data, count: Int(aacBuffer!.byteLength)) //has data
let aacReverseBuffer = AudioBufferConverter.convertToAAC(from: data) //has data
let pcmReverseBuffer = AudioBufferConverter.convertToPCM(from: aacBuffer2!, error: nil) //zeros data. data object exist, but filled by zeros

It's code for converting:

class AudioBufferFormatHelper {

    static func PCMFormat() -> AVAudioFormat? {

        return AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100, channels: 1, interleaved: false)
    }

    static func AACFormat() -> AVAudioFormat? {

        var outDesc = AudioStreamBasicDescription(
                mSampleRate: 44100,
                mFormatID: kAudioFormatMPEG4AAC,
                mFormatFlags: 0,
                mBytesPerPacket: 0,
                mFramesPerPacket: 0,
                mBytesPerFrame: 0,
                mChannelsPerFrame: 1,
                mBitsPerChannel: 0,
                mReserved: 0)
        let outFormat = AVAudioFormat(streamDescription: &outDesc)
        return outFormat
    }
}

class AudioBufferConverter {

    static func convertToAAC(from buffer: AVAudioBuffer, error outError: NSErrorPointer) -> AVAudioCompressedBuffer? {

        let outputFormat = AudioBufferFormatHelper.AACFormat()
        let outBuffer = AVAudioCompressedBuffer(format: outputFormat!, packetCapacity: 8, maximumPacketSize: 768)

        self.convert(from: buffer, to: outBuffer, error: outError)

        return outBuffer
    }

    static func convertToPCM(from buffer: AVAudioBuffer, error outError: NSErrorPointer) -> AVAudioPCMBuffer? {

        let outputFormat = AudioBufferFormatHelper.PCMFormat()
        guard let outBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat!, frameCapacity: 4410) else {
            return nil
        }

        outBuffer.frameLength = 4410
        self.convert(from: buffer, to: outBuffer, error: outError)

        return outBuffer
    }

    static func convertToAAC(from data: Data) -> AVAudioCompressedBuffer? {

        let nsData = NSData(data: data)
        let inputFormat = AudioBufferFormatHelper.AACFormat()
        let buffer = AVAudioCompressedBuffer(format: inputFormat!, packetCapacity: 8, maximumPacketSize: 768)
        buffer.byteLength = UInt32(data.count)
        buffer.packetCount = 8

        buffer.data.copyMemory(from: nsData.bytes, byteCount: nsData.length)
        buffer.packetDescriptions!.pointee.mDataByteSize = 4

        return buffer
    }

    private static func convert(from sourceBuffer: AVAudioBuffer, to destinationBuffer: AVAudioBuffer, error outError: NSErrorPointer) {

        //init converter
        let inputFormat = sourceBuffer.format
        let outputFormat = destinationBuffer.format
        let converter = AVAudioConverter(from: inputFormat, to: outputFormat)

        converter!.bitRate = 32000

        let inputBlock : AVAudioConverterInputBlock = { inNumPackets, outStatus in

            outStatus.pointee = AVAudioConverterInputStatus.haveData
            return sourceBuffer
        }

        _ = converter!.convert(to: destinationBuffer, error: outError, withInputFrom: inputBlock)
    }
}

In result AVAudioPCMBuffer has data with zeros. And in messages I see errors:

AACDecoder.cpp:192:Deserialize:  Unmatched number of channel elements in payload
AACDecoder.cpp:220:DecodeFrame:  Error deserializing packet
[ac] ACMP4AACBaseDecoder.cpp:1337:ProduceOutputBufferList: (0x14f81b840) Error decoding packet 1: err = -1, packet length: 0
AACDecoder.cpp:192:Deserialize:  Unmatched number of channel elements in payload
AACDecoder.cpp:220:DecodeFrame:  Error deserializing packet
[ac] ACMP4AACBaseDecoder.cpp:1337:ProduceOutputBufferList: (0x14f81b840) Error decoding packet 3: err = -1, packet length: 0
AACDecoder.cpp:192:Deserialize:  Unmatched number of channel elements in payload
AACDecoder.cpp:220:DecodeFrame:  Error deserializing packet
[ac] ACMP4AACBaseDecoder.cpp:1337:ProduceOutputBufferList: (0x14f81b840) Error decoding packet 5: err = -1, packet length: 0
AACDecoder.cpp:192:Deserialize:  Unmatched number of channel elements in payload
AACDecoder.cpp:220:DecodeFrame:  Error deserializing packet
[ac] ACMP4AACBaseDecoder.cpp:1337:ProduceOutputBufferList: (0x14f81b840) Error decoding packet 7: err = -1, packet length: 0

What are the dimensions of the vairable data? I'm just thinking that if you're passing a stereo audiofile to your parser then your `PCMFormat()` function will fail as it will try to return only one channel. — matiastofteby, Jul 25 '18 at 10:49

Rhythmic Fistman · Accepted Answer · 2018-07-24T13:21:34.853

There were a few problems with your attempt:

you're not setting the multiple packet descriptions when you convert data -> AVAudioCompressedBuffer. You need to create them, as AAC packets are of variable size. You can either copy them from the original AAC buffer, or parse them from your data by hand (ouch) or by using the AudioFileStream api.
you re-create your AVAudioConverters over and over again - once for each buffer, throwing away their state. e.g. the AAC encoder for its own personal reasons needs to add 2112 frames of silence before it can get around to reproducing your audio, so recreating the converter gets you a whole lot of silence.
you present the same buffer over and over to the AVAudioConverter's input block. You should only present each buffer once.
the bit rate of 32000 didn't work (for me)

That's all I can think of right now. Try the following modifications to your code instead which you now call like so:

(p.s. I changed some of the mono to stereo so I could play the round trip buffers on my mac, whose microphone input is strangely stereo - you might need to change it back)

(p.p.s there's obviously some kind of round trip / serialising/deserialising attempt going on here, but what exactly are you trying to do? do you want to stream AAC audio from one device to another? because it might be easier to let another API like AVPlayer play the resulting stream instead of dealing with the packets yourself)

let aacBuffer = AudioBufferConverter.convertToAAC(from: buffer, error: nil)!
let data = Data(bytes: aacBuffer.data, count: Int(aacBuffer.byteLength))
let packetDescriptions = Array(UnsafeBufferPointer(start: aacBuffer.packetDescriptions, count: Int(aacBuffer.packetCount)))
let aacReverseBuffer = AudioBufferConverter.convertToAAC(from: data, packetDescriptions: packetDescriptions)!
// was aacBuffer2
let pcmReverseBuffer = AudioBufferConverter.convertToPCM(from: aacReverseBuffer, error: nil)

class AudioBufferFormatHelper {

    static func PCMFormat() -> AVAudioFormat? {
        return AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100, channels: 1, interleaved: false)
    }

    static func AACFormat() -> AVAudioFormat? {

        var outDesc = AudioStreamBasicDescription(
            mSampleRate: 44100,
            mFormatID: kAudioFormatMPEG4AAC,
            mFormatFlags: 0,
            mBytesPerPacket: 0,
            mFramesPerPacket: 0,
            mBytesPerFrame: 0,
            mChannelsPerFrame: 1,
            mBitsPerChannel: 0,
            mReserved: 0)
        let outFormat = AVAudioFormat(streamDescription: &outDesc)
        return outFormat
    }
}

class AudioBufferConverter {
    static var lpcmToAACConverter: AVAudioConverter! = nil

    static func convertToAAC(from buffer: AVAudioBuffer, error outError: NSErrorPointer) -> AVAudioCompressedBuffer? {

        let outputFormat = AudioBufferFormatHelper.AACFormat()
        let outBuffer = AVAudioCompressedBuffer(format: outputFormat!, packetCapacity: 8, maximumPacketSize: 768)

        //init converter once
        if lpcmToAACConverter == nil {
            let inputFormat = buffer.format

            lpcmToAACConverter = AVAudioConverter(from: inputFormat, to: outputFormat!)
//            print("available rates \(lpcmToAACConverter.applicableEncodeBitRates)")
//          lpcmToAACConverter!.bitRate = 96000
            lpcmToAACConverter.bitRate = 32000    // have end of stream problems with this, not sure why
        }

        self.convert(withConverter:lpcmToAACConverter, from: buffer, to: outBuffer, error: outError)

        return outBuffer
    }

    static var aacToLPCMConverter: AVAudioConverter! = nil

    static func convertToPCM(from buffer: AVAudioBuffer, error outError: NSErrorPointer) -> AVAudioPCMBuffer? {

        let outputFormat = AudioBufferFormatHelper.PCMFormat()
        guard let outBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat!, frameCapacity: 4410) else {
            return nil
        }

        //init converter once
        if aacToLPCMConverter == nil {
            let inputFormat = buffer.format

            aacToLPCMConverter = AVAudioConverter(from: inputFormat, to: outputFormat!)
        }

        self.convert(withConverter: aacToLPCMConverter, from: buffer, to: outBuffer, error: outError)

        return outBuffer
    }

    static func convertToAAC(from data: Data, packetDescriptions: [AudioStreamPacketDescription]) -> AVAudioCompressedBuffer? {

        let nsData = NSData(data: data)
        let inputFormat = AudioBufferFormatHelper.AACFormat()
        let maximumPacketSize = packetDescriptions.map { $0.mDataByteSize }.max()!
        let buffer = AVAudioCompressedBuffer(format: inputFormat!, packetCapacity: AVAudioPacketCount(packetDescriptions.count), maximumPacketSize: Int(maximumPacketSize))
        buffer.byteLength = UInt32(data.count)
        buffer.packetCount = AVAudioPacketCount(packetDescriptions.count)

        buffer.data.copyMemory(from: nsData.bytes, byteCount: nsData.length)
        buffer.packetDescriptions!.pointee.mDataByteSize = UInt32(data.count)
        buffer.packetDescriptions!.initialize(from: packetDescriptions, count: packetDescriptions.count)

        return buffer
    }


    private static func convert(withConverter: AVAudioConverter, from sourceBuffer: AVAudioBuffer, to destinationBuffer: AVAudioBuffer, error outError: NSErrorPointer) {
        // input each buffer only once
        var newBufferAvailable = true

        let inputBlock : AVAudioConverterInputBlock = {
            inNumPackets, outStatus in
            if newBufferAvailable {
                outStatus.pointee = .haveData
                newBufferAvailable = false
                return sourceBuffer
            } else {
                outStatus.pointee = .noDataNow
                return nil
            }
        }

        let status = withConverter.convert(to: destinationBuffer, error: outError, withInputFrom: inputBlock)
        print("status: \(status.rawValue)")
    }
}

Hi! Thank you for you interest and for your answer! It's for stream audio from device to device, - AVPlayer and FileStreamApi won't help with this case. I found the solution and the main problem of my code above: incompatible settings for compressed buffer and for pcm buffer. Moreover, you are right, incorrect package size for compressed buffer. Bitrate 32000 will work, but with mono in my case. Why doesn't work in your code.. hm.. it's strange. Converter has a property with available bitrates, maybe setting for buffers gets other available bitrates. Later I will share my code and description. — Vladimir Prigarin, Jul 23 '18 at 00:48
In your code I seen incorrect size for compressed buffer. Buffer will be bigger then original PCM buffer - it's not good:) — Vladimir Prigarin, Jul 23 '18 at 00:51
I guess 32000 wasn't working because I was using a stereo microphone. Oops, yes the buffer size wasn't meant to be that big, I've changed it back to your original estimate of 8*768. — Rhythmic Fistman, Jul 23 '18 at 06:08
You _really_ need to 1. set the packet descriptions, 2. create the `AVAudioConverter`s once only, 3. input each buffer only once. Compatible settings can't fix those things. Have have you actually listened to your round-trip audio yet? — Rhythmic Fistman, Jul 23 '18 at 06:19
Yes. Only correct compatible settings fixes all problems. Yes, audio correct streaming on other device and back. — Vladimir Prigarin, Jul 23 '18 at 07:19
@RhythmicFistman Thanks for your answer. I am able to use it to create the aac buffer from data but, when I try to convert the aac buffer to a pcm buffer, the pcm buffer's data contains only 0s. Any ideas? — zoecarver, Aug 18 '18 at 20:05
Can you show how you're doing that? Maybe in a new question? — Rhythmic Fistman, Aug 18 '18 at 20:42
Was your compressed data smaller than the the original PCM data? In my case, if i set the bitrate to 92kbps, the compressed buffer is larger than the original. If I set the bitrate to 32kbbps, the compressed is smaller but only about 30% smaller. I assume i must be doing something right because if i stream the compressed buffer to another device, it does play. — Paul, Feb 24 '19 at 02:39
I _think_ this is ok, although I came across this code last week and thought the . `maximumPacketSize` shouldn't be a constant, it should be `lpcmToAACAudioConverter.maximumOutputPacketSize`. — Rhythmic Fistman, Feb 24 '19 at 06:59
In my case, I am getting aac audio buffer from the network in the form of NSData. So now I want to decode it into PCM. Please help. — Bhuvanendra Pratap Maurya, Sep 20 '19 at 11:20
Then ask a new question. If you create a new question you can add details about how you want the LPCM, e.g. in an array of floats, in a `CMSampleBuffer`, `AVAudioPCMBuffer`, an `AudioBufferList` or something else. — Rhythmic Fistman, Sep 20 '19 at 15:08
what is the fix of AACDecoder.cpp:189:Deserialize: Too few bits left in input buffer? which are coming randomly. — Bhuvanendra Pratap Maurya, Oct 29 '19 at 13:12
Is this something you're seeing with the code in this question or should this be a new question? — Rhythmic Fistman, Oct 29 '19 at 22:49

Decode AAC to PCM format using AVAudioConverter Swift

1 Answers1

Linked