Adding AVCaptureDeviceInput to CaptureSession resets/refocuses video

Question

A video recording app. I want it to work without stopping/pausing background music (when user listens to Apple Music for instance). This I can do nicely with setting category to mixWithOthers on AVAudioSession singleton.

After setting the category I also need to add AVCaptureDeviceInput to AVCaptureSession (so audio will get recorded). This results a glitch/hiccup to background audio and also video resets/refocuses.

I have investigated and it seems background audio glitch is something that can't be avoided, but video should not reset itself when input is added. The result of video resetting is that the first frame of the recorded video is either dark/black or it starts with out of focus frame and then focuses.

Also checked Snapchat ios app and they also have audio glitch when starting recording, but video starts to record smoothly. What am I doing wrong.

My code:

//Setting audio session to mixWithOthers upon startup
let session = AVAudioSession.sharedInstance()

do {
    try session.setCategory(AVAudioSessionCategoryPlayAndRecord,
                            with: [.mixWithOthers])
    if session.mode != AVAudioSessionModeVideoRecording {
        try session.setMode(AVAudioSessionModeVideoRecording)
    }
} catch let error {
    print("avsession category error: \(error)")
}

And then:

//Just before recording starts will add audio input
let audioDevice = AVCaptureDevice.defaultDevice(withMediaType: AVMediaTypeAudio)
do
{
    let deviceInput = try AVCaptureDeviceInput(device: audioDevice) as AVCaptureDeviceInput
    if captureSession.canAddInput(deviceInput) {
        captureSession.addInput(deviceInput)
    }
    else {
        print("Could not add audio device input to the session")
    }


}
catch let error as NSError {
    print(error.localizedDescription)
    return
}

Would it be possible to do this without any glitches at all? If not then how could I make it at least like Snapchat (no video reset upon addInput call)?

Answer 1

I've noticed AVCaptureSession addInput resets silence set by AVAudioSession AVAudioSessionCategory(Solo)Ambient category

 Category                              Silenced   Interrupts
 AVAudioSessionCategoryAmbient         Yes        No
 AVAudioSessionCategorySoloAmbient     Yes        Yes
 AVAudioSessionCategoryMultiRoute      No         Yes
 AVAudioSessionCategoryPlayAndRecord   No         Yes
 AVAudioSessionCategoryPlayback        No         Yes
 AVAudioSessionCategoryRecord          No         Yes

https://developer.apple.com/library/content/documentation/Audio/Conceptual/AudioSessionProgrammingGuide/AudioSessionCategoriesandModes/AudioSessionCategoriesandModes.html

When the user moves the Silent switch (or Ring/Silent switch on iPhone) to the “silent” position, your audio is silenced.

AVAudioSessionCategoryAmbient—Playback only. Plays sounds that add polish or interest but are not essential to the app’s use. Using this category, your audio is silenced by the Ring/Silent switch and when the screen locks.

AVAudioSessionCategorySoloAmbient—(Default) Playback only. Silences audio when the user switches the Ring/Silent switch to the “silent” position and when the screen locks. This category differs from the AVAudioSessionCategoryAmbient category only in that it interrupts other audio.

Answer 2

So from your question, this sentence specifically:

The result of video resetting is that the first frame of the recorded video is either dark/black or it starts with out of focus frame and then focuses.

Indicates to me you are adding the audioInput after having begun video recording. Is it possible to add the audioInput device before video begins recording? I imagine it's expected behavior for the video to "freeze" when adding another input source, since the AVCaptureSession essentially just takes inputs and outputs a file, more or less. I'd imagine that adding the audioInput device before recording would solve that issue in the final product, since that action would not end up in the recording! :)

Another comment that was mentioned was:

Are you calling beginConfiguration / commitConfiguration on the session before changing inputs ?

– SeanLintern88 Apr 10 at 15:30

I think this was not addressed by the question author, and could also be the cause of issues. As shown in this question/answer, you need to add the audioInput device between a begin/commit configuration call.

In general, I find the core Apple docs very helpful, and will probably have more info for more things you'll likely want to do.