How to add multiple audio AVCaptureDevice to an AVCaptureSession

Question

I am trying to add multiple audio AVCaptureDeviceInputs to an AVCaptureSession (macOS).

My understanding is that AVCaptureConnections are automatically created between compatible inputs and outputs.

I can successfully add a single camera and audio device absolutely fine. My problem comes when adding a second audio device. I would like to overlay the audio from multiple sources.

Audio inputs are added as follows:

for device in newDevices {
   do {
      let newAudioDeviceInput = try AVCaptureDeviceInput(device: device)
      if self.captureSession.canAddInput(newAudioDeviceInput) {
         self.captureSession.addInput(newAudioDeviceInput)
         self.audioDeviceInputs.append(newAudioDeviceInput)
      }
   } catch {
      errors.append(error)
   }
}

print(captureSession.inputs.count) successfully shows the correct number of inputs

Output is added as follows:

let audioDataOutput = AVCaptureAudioDataOutput()
audioDataOutput.setSampleBufferDelegate(delegate, queue: queue)
if self.captureSession.canAddOutput(audioDataOutput) {
   self.captureSession.addOutput(audioDataOutput)
   self.audioDataOutput = audioDataOutput
} else {
   throw Error.cannotAddOutput
}

A single AVCaptureConnection is created between the first input added and the output, but no subsequent inputs are added to the connection.

I'm really struggling to find any decent documentation on this, so any suggestions much appreciated. My expectation would be each additional AVCaptureDeviceInput would be added to the same connection inputs array.

Rhythmic Fistman · Accepted Answer · 2021-11-15T11:35:39.883

I thought multiple audio and video AVCaptureInputs were unsupported, yet I couldn't find any documentation on that either.

A while ago this person cleverly managed to create multiple video inputs and outputs by using addOutputWithNoConnections(), then manually creating the connections to the inputs (but why no addInputWithNoConnections()?):

https://stackoverflow.com/a/30191013/22147

That could be worth trying with audio! Please report back with your results if you try this!

However I like to wrap the input devices in an aggregate audio device and then configure the AVCaptureSession to use that. This gives you the convenience of working with a single "device" and you don't have to worry about multiple clocks and timestamps because both devices are synchronised.

You can create an aggregate audio device manually in Audio MIDI Setup.app: https://stackoverflow.com/a/65704755/22147

or programmatically, using AudioHardwareCreateAggregateDevice(): https://stackoverflow.com/a/56415699/22147 If you use the programmatic route you can hide the resulting aggregate device by setting kAudioAggregateDeviceIsPrivateKey to true. You might want to do this to stop users messing with it.

Thanks for the great reply. Interestingly I'd just stumbled over the video inputs solution this morning, so might give that a go and see if it works. That said, I do really like the idea of creating an aggregate device as it would potentially give a lot more control. I'll report back. — DanM, Nov 15 '21 at 11:25

How to add multiple audio AVCaptureDevice to an AVCaptureSession

1 Answers1