How to delegate ARSession and AVCaptureSession at the same time?

Question

I've seen people running ARKit with ARSCNView, and they're able to fetch said a QR code rectangle in the camera and render something at the position somehow related to the place QR code resided in the space. I thought they did it by delegating AVCaptureMetadataOutputObjectsDelegate, so I went ahead and attempt to achieve the following logic:

Running ARSCNViewDelegate, ARSessionDelegate and AVCaptureMetadataOutputObjectsDelegate at the same time.
Wire up an ARSCNView, delegate the view and its session to self.
Wire up an AVCaptureSession, delegate it to self, and start the session.
Start the ARSCNView session, since the official example did this in viewWillAppear(), I did the same thing.

The application did run without any error, but I could only get a few callbacks from AVCaptureMetadataOutputObjectsDelegate, like 3 to 5, and then it would never be called again like the delegate of ARSession seized its authority, was I implementing the wrong approach or you just can't delegate ARSession and AVCaptureSession at the same time?

Did you know what they use to render something at the 3D position of the QRCode ? — Alain Berrier, Aug 01 '17 at 12:18
If you're able to get the metadata of the camera frame, you can transfer the rect of QRCode to 3D coordinate and do your render stuff. — Cytus Chang, Aug 02 '17 at 09:37
This is exactly what I was trying to do a week ago, but I didn't succeed...I'm trying to understand you can transfer the rect of QRCode knowing its position on screen to 3D coordinate.Do you need a particular framework to do that? — Alain Berrier, Aug 02 '17 at 10:40
Check https://stackoverflow.com/questions/44579839/ios-revert-camera-projection. — Cytus Chang, Aug 04 '17 at 01:49

score 7 · Accepted Answer · edited Jan 13 '21 at 15:21

I did exactly what you describe and have the same problems. It seems that ARKit relies upon the AVCapture system, and doesn't support using more than one capture device at a time. I found two solutions.

Start an ARSession, and implement the function session(_:didUpdate:) of ARSessionDelegate. Every time you capture an ARFrame search for a QR code in the image of the frame. (code)

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    let image = CIImage(cvPixelBuffer: frame.capturedImage)
    let detector = CIDetector(ofType: CIDetectorTypeQRCode, context: nil, options: nil)
    let features = detector!.features(in: image)

    for feature in features as! [CIQRCodeFeature] {
        if !discoveredQRCodes.contains(feature.messageString!) {
            discoveredQRCodes.append(feature.messageString!)
            let url = URL(string: feature.messageString!)
            let position = SCNVector3(frame.camera.transform.columns.3.x,
                                      frame.camera.transform.columns.3.y,
                                      frame.camera.transform.columns.3.z)
        }
     }
}

Start an AVCaptureSession, when you identify and decode your QR code stop it and start an ARSession. (not recommended)

How to delegate ARSession and AVCaptureSession at the same time?

1 Answers1