How can I move a model using facial expression like look-right?

Question

I have set a virtual object in rear camera view. I want to move that object using facial expression with respect to world origin and measure the displacement angles of the virtual object.

Is that possible using ARKit or RealityKit?

Answer 1

Use the following solution. At first setup a configuration:

import RealityKit
import ARKit

class ViewController: UIViewController, ARSessionDelegate {

    @IBOutlet var arView: ARView!

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        arView.session.delegate = self
        arView.automaticallyConfigureSession = false
       
        let config = ARFaceTrackingConfiguration()
        config.isWorldTrackingEnabled = true        // Simultaneous tracking
        arView.session.run(config)
    }
}

Run your transform animation when a defined facial expression occurs:

func facialExpression(anchor: ARFaceAnchor) {
    
    let eyeUpLeft = anchor.blendShapes[.eyeLookUpLeft]
    let eyeUpRight = anchor.blendShapes[.eyeLookUpRight]

    if ((eyeUpLeft?.decimalValue ?? 0.0) + 
        (eyeUpRight?.decimalValue ?? 0.0)) > 0.75 {

        // ModelEntity's animation goes here
    }
}

Delegate's method (running at 60 fps):

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {

    guard let faceAnchor = anchors[0] as? ARFaceAnchor else { return }
    self.facialExpression(anchor: faceAnchor)
}

The answer to your second question you can see HERE.