RemoteIO AudioUnit playback quality not tied to callback runtime, but to something else

Question

Playback through my AudioUnit works fine until I start getting gyroscope updates from a CMMotionManager. I assumed this was due to a performance hit, but when I measured the runtime of my callback during said gyroscope updates it isn't as high as other CMMotionManager-less trials with smooth playback, yet the playback is choppy.

Some visual explanation (The red is the time between consecutive callbacks. The green--it's hard to see but there's bits of green right underneath all the red--is the runtime of the callback, which is consistently just a few milliseconds less):

graph

Sorry if the graph is a bit messy, hopefully I'm still getting my point across.

In sum, rather than the runtime of the callback, the quality of the playback seems more tied to the "steadiness" of the frequency at which the callback is, erm, called back. What could be going on here? Could my callback runtimes just be off? That seems unlikely. I'm timing my callback via calls to clock() at the beginning and end. Is my AudioUnit setup wrong? It is admittedly a bit hacked together, and I'm not using an AUGraph or anything.

AudioUnit initialization code:

AudioComponentDescription desc;

    desc.componentType = kAudioUnitType_Output;
    desc.componentSubType = kAudioUnitSubType_RemoteIO; // Remote I/O is for talking with the hardware
    desc.componentFlags = 0;
    desc.componentFlagsMask = 0;
    desc.componentManufacturer = kAudioUnitManufacturer_Apple;

    AudioComponent component = AudioComponentFindNext(NULL, &desc);
    AudioComponentInstanceNew(component, &myAudioUnit);

    UInt32 enableIO;
    AudioUnitElement inputBus = 1;
    AudioUnitElement outputBus = 0;

    //Disabling IO for recording
    enableIO = 0;
    AudioUnitSetProperty(myAudioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, inputBus, &enableIO, sizeof(enableIO));

    //Enabling IO for playback
    enableIO = 1;
    AudioUnitSetProperty(myAudioUnit, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Output, outputBus, &enableIO, sizeof(enableIO));

    UInt32 bytesPerSample = BIT_DEPTH/8.0;
    AudioStreamBasicDescription stereoStreamFormat = {0};
    stereoStreamFormat.mBitsPerChannel = 8 * bytesPerSample;
    stereoStreamFormat.mBytesPerFrame = bytesPerSample;
    stereoStreamFormat.mBytesPerPacket = bytesPerSample;
    stereoStreamFormat.mChannelsPerFrame = 2;   // 2 indicates stereo
    stereoStreamFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger |
    kAudioFormatFlagsNativeEndian |
    kAudioFormatFlagIsPacked |
    kAudioFormatFlagIsNonInterleaved;
    stereoStreamFormat.mFormatID = kAudioFormatLinearPCM;
    stereoStreamFormat.mFramesPerPacket = 1;
    stereoStreamFormat.mReserved = 0;
    stereoStreamFormat.mSampleRate = SAMPLE_RATE;

    AudioUnitSetProperty(myAudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, inputBus, &stereoStreamFormat, sizeof(AudioStreamBasicDescription));
    AudioUnitSetProperty(myAudioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, outputBus, &stereoStreamFormat, sizeof(AudioStreamBasicDescription));

    //Setting input callback
    AURenderCallbackStruct callbackStruct;
    callbackStruct.inputProc = &recordingCallback;    //TODO: Should there be an ampersand?
    callbackStruct.inputProcRefCon = myAudioUnit;
    AudioUnitSetProperty(myAudioUnit, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Output, inputBus, &callbackStruct, sizeof(callbackStruct));   //TODO: Not sure of scope and bus/element

    //Setting output callback
    callbackStruct.inputProc = &playbackCallback;
    callbackStruct.inputProcRefCon = myAudioUnit;
    AudioUnitSetProperty(myAudioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Input, outputBus, &callbackStruct, sizeof(callbackStruct));
    AudioUnitInitialize(myAudioUnit);

RemoteIO Playback Callback:

static OSStatus playbackCallback (void *inRefCon, AudioUnitRenderActionFlags *ioActionFlags, const AudioTimeStamp *inTimeStamp, UInt32 inBusNumber, UInt32 inNumberFrames, AudioBufferList *ioData) {

    timeOfCallback = clock();
    if (timeOfPrevCallback != 0) {
        callbackInterval = (double)timeOfCallback - timeOfPrevCallback;
    }
    clock_t t1, t2;
    t1 = clock();

    FooBarClass::myCallbackFunction((short *)ioData->mBuffers[0].mData, (short *)ioData->mBuffers[1].mData);
    t2 = clock();

    cout << "Callback duration: " << ((double)(t2-t1))/CLOCKS_PER_SEC << endl;
    cout << "Callback interval: " << callbackInterval/CLOCKS_PER_SEC << endl;

    timeOfPrevCallback = timeOfCallback;
    //prevCallbackInterval = callbackInterval;
    return noErr;
}

In myCallbackFunction I'm reading from a handful of .wav files, filtering each one and mixing them together, and copying the output to the buffers passed to it. In the graph where I mention "incrementally adding computation" I'm referring to the number of input files I'm mixing together. Also, if it matters, gyroscope updates occur via an NSTimer that goes off every 1/25 of a second:

[self.getMotionManager startDeviceMotionUpdates];
    gyroUpdateTimer = [NSTimer scheduledTimerWithTimeInterval:GYRO_UPDATE_INTERVAL target:self selector:@selector(doGyroUpdate) userInfo:nil repeats:YES];
...
+(void)doGyroUpdate {
    double currentYaw = motionManager.deviceMotion.attitude.yaw;
    // a couple more lines of not very expensive code
}

I should also be more clear about what I mean by choppiness in this sense: The audio isn't skipping, it just sounds really bad, as if an additional, crackly track was getting mixed in while the other tracks play smoothly. I'm not talking about clipping either, which it isn't because the only difference between good and bad playback is the gyroscope updates.

Thanks in advance for any tips.

----- Update 1 -----

My runtimes were a bit off because I was using clock(), which apparently doesn't work right for multithreaded applications. Apparently get_clock_time() is the proper way to measure runtimes across multiple threads but it's not implemented for Darwin. Though it's not an ideal solution, I'm using gettimeofday() now to measure the callback run time and intervals. Aside from the now steady intervals between successive callbacks (which were previously pretty erratic), things are more or less the same:

updated graph

----- Update 2 -----

Interestingly enough, when I start and then stop CMMotionManager updates altogether via stopDeviceMotionUpdates, the crackliness persists...

----- Update 3 -----

'Crackliness' doesn't start until the first CMMotionManager is received, i.e. when the deviceMotion property is first checked after the NSTimer is first triggered. After that, crackliness persists regardless of the update frequency and even after updates are stopped.

You're using Objective-C in a callback which is generally not a real -time safe thing to do. It may or may not be the source of the problem but it isn't a good idea. — sbooth, Jan 15 '15 at 03:01
Yeah that's a good point. I changed it but it's just a wrapper to a C++ call so it unfortunately didn't make any difference. — acannon828, Jan 15 '15 at 03:59
My guess is that the frequency of gyroscope updates is hogging the processor and starving other threads. If you try 5 updates per second does the audio work properly? — sbooth, Jan 15 '15 at 13:06
Nope. Nor if I update at 1Hz. It doesn't even sound any better. Does clock()-ing measure CPU cycles on the current thread or across all threads? My assumption is that it's the latter, in which case the runtimes in my graph above are correct, and indicate that though there is a performance hit when gyroscope updates start, the callback is executing quickly enough for smooth playback. — acannon828, Jan 15 '15 at 16:24
Wait, [clock()-ing only measures CPU cycles on the current thread](http://stackoverflow.com/questions/2962785/c-using-clock-to-measure-time-in-multi-threaded-programs), so my runtimes may be off. I'll fix that and see what happens... — acannon828, Jan 15 '15 at 16:27

hotpaw2 · Answer 1 · 2016-06-19T00:23:38.017

You are trying to call Objective C methods, do synchronous file reads, and/or do significant computation (your C++ function) inside a real-time audio render callback. Also, logging to cout from inside a real-time thread is most likely not going to work reliably. These operations can take too long to meet the latency critical real-time requirements of Audio Unit callbacks.

Instead, for any data that does not have a tightly bounded maximum latency to generate, you might just copy that data from a lock free circular fifo or queue inside your render callback, and fill that audio data fifo slightly ahead of time in another thread (perhaps running on an NSTimer or CADisplayLink).

score 0 · Answer 2 · answered Mar 30 '15 at 05:15

I had a similar issue when activating the location services. The issue was only present on slower devices like the iPod touch 16gb and not present on other hardware. I saw that you have in your graph title BUF_SIZE: 1024 Is this your call back time? I fixed my problem by increasing the callback time (buffer size). If you can handle more latency, try increasing the callback time using

NSTimeInterval _preferredDuration = (2048) / 44100.0 ;  // Try bigger values here
NSError* err;
[[AVAudioSession sharedInstance]setPreferredIOBufferDuration:_preferredDuration error:&err];

RemoteIO AudioUnit playback quality not tied to callback runtime, but to something else

----- Update 1 -----

----- Update 2 -----

----- Update 3 -----

2 Answers2