Low latency input/output AudioQueue

Question

I have two iOS AudioQueues - one input that feeds samples directly to one output. Unfortunately, there is an echo effect that is quite noticeable :(

Is it possible to do low latency audio using AudioQueues or do I really need to use AudioUnits? (I have tried the Novocaine framework that uses AudioUnits and here the latency is much smaller. I've also noticed this framework seems to use less CPU resources. Unfortunately, I was unable to use this framework in my Swift project without major changes to it.)

Here are some extracts of my code, which is mainly done in Swift, except those callbacks which needs to be implemented in C.

private let audioStreamBasicDescription = AudioStreamBasicDescription(
    mSampleRate: 16000,
    mFormatID: AudioFormatID(kAudioFormatLinearPCM),
    mFormatFlags: AudioFormatFlags(kAudioFormatFlagsNativeFloatPacked),
    mBytesPerPacket: 4,
    mFramesPerPacket: 1,
    mBytesPerFrame: 4,
    mChannelsPerFrame: 1,
    mBitsPerChannel: 32,
    mReserved: 0)

private let numberOfBuffers = 80
private let bufferSize: UInt32 = 256

private var active = false

private var inputQueue: AudioQueueRef = nil
private var outputQueue: AudioQueueRef = nil

private var inputBuffers = [AudioQueueBufferRef]()
private var outputBuffers = [AudioQueueBufferRef]()
private var headOfFreeOutputBuffers: AudioQueueBufferRef = nil

// callbacks implemented in Swift
private func audioQueueInputCallback(inputBuffer: AudioQueueBufferRef) {
    if active {
        if headOfFreeOutputBuffers != nil {
            let outputBuffer = headOfFreeOutputBuffers
            headOfFreeOutputBuffers = AudioQueueBufferRef(outputBuffer.memory.mUserData)
            outputBuffer.memory.mAudioDataByteSize = inputBuffer.memory.mAudioDataByteSize
            memcpy(outputBuffer.memory.mAudioData, inputBuffer.memory.mAudioData, Int(inputBuffer.memory.mAudioDataByteSize))
            assert(AudioQueueEnqueueBuffer(outputQueue, outputBuffer, 0, nil) == 0)
        } else {
            println(__FUNCTION__ + ": out-of-output-buffers!")
        }

        assert(AudioQueueEnqueueBuffer(inputQueue, inputBuffer, 0, nil) == 0)
    }
}

private func audioQueueOutputCallback(outputBuffer: AudioQueueBufferRef) {
    if active {
        outputBuffer.memory.mUserData = UnsafeMutablePointer<Void>(headOfFreeOutputBuffers)
        headOfFreeOutputBuffers = outputBuffer
    }
}

func start() {
    var error: NSError?
    audioSession.setCategory(AVAudioSessionCategoryPlayAndRecord, withOptions: .allZeros, error: &error)
    dumpError(error, functionName: "AVAudioSessionCategoryPlayAndRecord")
    audioSession.setPreferredSampleRate(16000, error: &error)
    dumpError(error, functionName: "setPreferredSampleRate")
    audioSession.setPreferredIOBufferDuration(0.005, error: &error)
    dumpError(error, functionName: "setPreferredIOBufferDuration")

    audioSession.setActive(true, error: &error)
    dumpError(error, functionName: "setActive(true)")

    assert(active == false)
    active = true

    // cannot provide callbacks to AudioQueueNewInput/AudioQueueNewOutput from Swift and so need to interface C functions
    assert(MyAudioQueueConfigureInputQueueAndCallback(audioStreamBasicDescription, &inputQueue, audioQueueInputCallback) == 0)
    assert(MyAudioQueueConfigureOutputQueueAndCallback(audioStreamBasicDescription, &outputQueue, audioQueueOutputCallback) == 0)

    for (var i = 0; i < numberOfBuffers; i++) {
        var audioQueueBufferRef: AudioQueueBufferRef = nil
        assert(AudioQueueAllocateBuffer(inputQueue, bufferSize, &audioQueueBufferRef) == 0)
        assert(AudioQueueEnqueueBuffer(inputQueue, audioQueueBufferRef, 0, nil) == 0)
        inputBuffers.append(audioQueueBufferRef)

        assert(AudioQueueAllocateBuffer(outputQueue, bufferSize, &audioQueueBufferRef) == 0)
        outputBuffers.append(audioQueueBufferRef)

        audioQueueBufferRef.memory.mUserData = UnsafeMutablePointer<Void>(headOfFreeOutputBuffers)
        headOfFreeOutputBuffers = audioQueueBufferRef
    }

    assert(AudioQueueStart(inputQueue, nil) == 0)
    assert(AudioQueueStart(outputQueue, nil) == 0)
}

And then my C-code to set up the callbacks back to Swift:

static void MyAudioQueueAudioInputCallback(void * inUserData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer, const AudioTimeStamp * inStartTime,
                                   UInt32 inNumberPacketDescriptions, const AudioStreamPacketDescription * inPacketDescs) {
    void(^block)(AudioQueueBufferRef) = (__bridge void(^)(AudioQueueBufferRef))inUserData;
    block(inBuffer);
}

static void MyAudioQueueAudioOutputCallback(void *inUserData, AudioQueueRef inAQ, AudioQueueBufferRef inBuffer) {
    void(^block)(AudioQueueBufferRef) = (__bridge void(^)(AudioQueueBufferRef))inUserData;
    block(inBuffer);
}

OSStatus MyAudioQueueConfigureInputQueueAndCallback(AudioStreamBasicDescription inFormat, AudioQueueRef *inAQ, void(^callback)(AudioQueueBufferRef)) {
    return AudioQueueNewInput(&inFormat, MyAudioQueueAudioInputCallback, (__bridge_retained void *)([callback copy]), nil, nil, 0, inAQ);
}

OSStatus MyAudioQueueConfigureOutputQueueAndCallback(AudioStreamBasicDescription inFormat, AudioQueueRef *inAQ, void(^callback)(AudioQueueBufferRef)) {
    return AudioQueueNewOutput(&inFormat, MyAudioQueueAudioOutputCallback, (__bridge_retained void *)([callback copy]), nil, nil, 0, inAQ);
}

score 2 · Accepted Answer · edited May 23 '17 at 12:01

2

After a good while I found this great post using AudioUnits instead of AudioQueues. I just ported it to Swift and then simply added:

audioSession.setPreferredIOBufferDuration(0.005, error: &error)

edited May 23 '17 at 12:01

Community

1
1

answered May 29 '15 at 14:20

Jens Schwarzer

2,840
1
22
35

Could you share the Swift version somewhere? – Joel May 09 '16 at 20:01
Did it also solve the latency problem? What about CPU resources? – rsp1984 Jul 04 '16 at 22:50
@rsp1984 I've looked at my code and found this comment: "a value of 5 ms seems to introduce ~1% of CPU usage on iPhone 5". Then I recalled if I decreased it further the CPU usage began to increase more. And of course the latency cannot be eliminated but it was reduced to an acceptable level :) BTW: The latency on iPod Touch is greater than for example on iPhone 5. – Jens Schwarzer Jul 05 '16 at 07:27
@Jens Schwarzer Thanks much! – rsp1984 Jul 05 '16 at 20:22

score 1 · Answer 2 · answered May 04 '15 at 00:20

If you're recording audio from a microphone and playing it back within earshot of that microphone, then due to the audio throughput not being instantaneous, some of your previous output will make it into the new input, hence the echo. This phenomenon is called feedback.

This is a structural problem, so changing the recording API won't help (although changing your recording/playback buffer sizes will give you control over the delay in the echo). You can either play back the audio in such a way that the microphone can't hear it (e.g. not at all, or through headphones) or go down the rabbit hole of echo cancellation.

Hi and thanks for your input. No, it is not feedback I have problems with. I am using headphones ;) And as mentioned in my question I don't get a latency problem (only very minor) if I use the Novocaine framework. — Jens Schwarzer, May 04 '15 at 14:20

Low latency input/output AudioQueue

2 Answers2

Linked