Error Capturing DepthData using AVCapturePhotoOutput

Question

I'm trying to take a photo with depth data using the .builtInDualCamera. I've been following the Apple example, AVCamFilter (which works for me). From what I understood from a WWDC presentation, all you need to do is set up an AVCapturePhotoOutput to enable depth data capture and enable same on the AVCapturePhotoSettings.

When I run my app, I get a generic error (see below).

If I remove the set up for capturing depth data, no error occurs and a photo is captured.
If I leave the set up for capturing depth data intact BUT I set a break point in the AVCapturePhotoCaptureDelegate function, photoOutput(_:AVCapturePhotoOutput, didCapturePhotoFor: AVCaptureResolvedPhotoSettings), when I take the photo and the app stops at this break point and I immediately continue running, the photo + depth data is captured (no error happens).
The iPhone I have has a Dual camera and photoOutput.isDepthDataDeliverySupported is true
iPhone 12 Pro Max; iOS 15.2
Xcode 13.2.1
macOS 11.6.2

I have no idea what I am missing.

Here is my code to set up the inputs and outputs in the AVCaptureSession:

guard let videoDevice = discoverDevice(from: [.builtInDualCamera]) else {
            fatalError("No dual camera.")
        }
guard let videoDeviceInput = try? AVCaptureDeviceInput(device: videoDevice) else {
    fatalError("Can't create video input.")
}
        
self.session.beginConfiguration()
        
self.session.sessionPreset = .photo
        
guard self.session.canAddInput(videoDeviceInput) else {
    fatalError("Can't add video input.")
}
self.session.addInput(videoDeviceInput)

guard self.session.canAddOutput(photoOutput) else {
    fatalError("Can't add photo output.")
}
self.session.addOutput(photoOutput)
        
photoOutput.isHighResolutionCaptureEnabled = true
        
if photoOutput.isDepthDataDeliverySupported {
    photoOutput.isDepthDataDeliveryEnabled = true
} else {
    fatalError("DepthData is not supported by this camera configuration")
}
        
self.session.commitConfiguration()
        
self.videoDeviceInput = videoDeviceInput

this is the code called when I want to snap a photo (taken from AVCamFilter example):

sessionQueue.async {
    let photoSettings = AVCapturePhotoSettings(format: [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)])
    if self.photoOutput.isDepthDataDeliveryEnabled {
        photoSettings.isDepthDataDeliveryEnabled = true
        photoSettings.embedsDepthDataInPhoto = false
    }
        
    self.photoOutput.capturePhoto(with: photoSettings, delegate: self)
}

This is the error I am getting in the AVCapturePhotoCaptureDelegate function, photoOutput(_:AVCapturePhotoOutput, didFinishProcessingPhoto: AVCapturePhoto, error: Error?):

Error capturing photo: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-16800), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2829f9650 {Error Domain=NSOSStatusErrorDomain Code=-16800 "(null)"}}

score 1 · Answer 1 · answered Dec 29 '21 at 21:13

I think I figured it out. It has to do with frame rates on the input device vs at what rate depth data can be captured. I used this function just before committing the session changes when configuring the input and outputs.

    private func capFrameRate(videoDevice: AVCaptureDevice) {
        if self.photoOutput.isDepthDataDeliverySupported {
            // Cap the video framerate at the max depth framerate.
            if let frameDuration = videoDevice.activeDepthDataFormat?.videoSupportedFrameRateRanges.first?.minFrameDuration {
                do {
                    try videoDevice.lockForConfiguration()
                    videoDevice.activeVideoMinFrameDuration = frameDuration
                    videoDevice.unlockForConfiguration()
                } catch {
                    print("Could not lock device for configuration: \(error)")
                }
            }
        }
    }

Once I did this I was able to get both the pixel buffer of the image and the depthData with it.

Which now makes sense(ish) when I used a breakpoint in the middle of the capture session, it must have acted as a sync for the frame rate. Whether or not that's the case, setting the activeVideoMinFrameDuration to make the depthData's rate was key.

Error Capturing DepthData using AVCapturePhotoOutput

1 Answers1