How to retrieve AVCameraCalibrationData from photo output on iOS?

Question

I'm looking to retrieve the lens distortion coefficients (note: not focal length nor principal point) associated with a photo that I capture on an iOS device. From my understanding, the only way to do so on a iOS device is to use AVCameraCalibrationData. The official documentation only provides information on how to retrieve the camera calibration data from AVDepthData, but both the documentation and this StackOverflow answer imply that AVCameraCalibrationData can be used with images, not just with depth data.

Is it possible to retrieve AVCameraCalibrationData information when you're capturing an image? If so, is there documentation around this functionality?

Answer 1

Background: A lot of these stack overflow responses are referencing intrinsic data when asked about camera calibration, but calibration data typically includes intrinsic data, extrinsic data, lens distortion, etc. Its all listed out here in the iOS documentation you linked to.

I am going to assume you have the general camera app code in place. In that code, when a picture is taken, you are probably going to make a call to the photoOutput function that looks likes something like this:

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {...

The output parameter is going to have a value you can reference to see if camera calibration is supported called isCameraCalibrationDataDeliverySupported, so for example, to print that out, use something like this:

print("isCameraCalibrationDataDeliverySupported: \(output.isCameraCalibrationDataDeliverySupported)")

Note in the documentation I linked to, it is only supported in specific scenarios:

"This property's value can be true only when the isDualCameraDualPhotoDeliveryEnabled property is true. To enable camera calibration delivery, set the isCameraCalibrationDataDeliveryEnabled property in a photo settings object."

So that's important, pay attention to that to avoid unnecessary stress. Use the actual value to debug and make sure you have the proper environment enabled.

With all that in place, you should get the actual camera calibration data from:

photo.cameraCalibrationData

Just pull out of that object to get specific values you are looking for, such as:

photo.cameraCalibrationData?.extrinsicMatrix
photo.cameraCalibrationData?.intrinsicMatrix
photo.cameraCalibrationData?.lensDistortionCenter
etc.

Basically everything that is listed in the documentation you linked to and that I linked to again.

One more thing to note, if you ARE looking for just the intrinsic matrix, that can be obtained much easier (i.e. not as stringent of an environment) than the rest of these values through the approach outlined in the the stack overflow. If you are using this for computer vision, which is what I am using it for, that is sometimes all that is needed. But for really cool stuff, you'll want it all.

Answer 2

You can get AVCameraCalibrationData only from depth data output or photo output.

However, if all you need is FOV, you need only part of the info that class offers — the camera intrinsics matrix — and you can get that by itself from AVCaptureVideoDataOutput.

Set cameraIntrinsicMatrixDeliveryEnabled on the AVCaptureConnection connecting your camera device to the capture session. (Note you should check cameraIntrinsicMatrixDeliverySupported first; not all capture formats support intrinsics.)

When the video output vends sample buffers, check each sample buffer's attachments for the kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix key. As noted in CMSampleBuffer.h (someone should file a radar about getting this info into the online documentation), the value for that attachment is a CFData encoding a matrix_float3x3, and the (0,0) and (1,1) elements of that matrix are the horizontal and vertical focal length .