0

I am using ARKit's ARFaceTrackingConfiguration to track the facial blendshapes along with left and right Eye Transforms. I am exporting this data into json and apply this data on 3d model ( which preconfigured shape keys, eye nodes). I was able to apply the blend shape data, but I got struck at how to apply the eye rotations. am getting leftEyeTransform, rightEyeTransform which is simd_float4*4 from FaceAnchor.

Here how to apply the rotation on eye nodes from the transform values.I believe for eyes, it is enough to apply the rotation.

I have tried with the below to get the orientation from eyeTransforms:

Method 1:

let faceNode = SCNNode()
faceNode.simdTransform = eyeTransform
let vector = faceNode.eulerAngles
eyeLeftNode.eulerAngles = vector

Method:2

let faceNode = SCNNode()
faceNode.simdTransform = eyeTransform
let rotation = vector_float3(faceNode.orientation.x,faceNode.orientation.y,faceNode.orientaton.z)
let yaw = (rotation.y)
let pitch = (rotation.x)
let roll = (rotation.z)
let vector = SCNVector3(pitch, yaw, roll)

eyeLeftNode.eulerAngles = vector

Method: 3

let simd_quatf = simd_quaternion(eyeTransform)
let vector = SCNVector3(simd_quatf.axis.x,simd_quatf.axis.y,simd_quatf.axis.z)

eyeLeftNode.eulerAngles = vector

None of the ways are working. I am not able to figure out the actual problem on how to rotate the eyeBalls. Can you please tell me how to do this

Thanks, Chaitanya

Chaitu
  • 907
  • 2
  • 13
  • 27

1 Answers1

1

I use the following two extensions in my apps for simd_float4x4 translation and orientation components if that's all you need:

extension float4x4 {
    var translation: SIMD3<Float> {
        let translation = columns.3
        return SIMD3<Float>(translation.x, translation.y, translation.z)
    }

    /**
     Factors out the orientation component of the transform.
    */
    var orientation: simd_quatf {
        return simd_quaternion(self)
    }
}
jfriesenhahn
  • 195
  • 10
  • Thank you for your reply. I am taking the orientation from the above method. and taking the axis and applied the euler angle. let leftSimdQuartf = faceAnchor.leftEyeTransform.orientation let leftEyeEuler = SCNVector3(leftSimdQuartf.x,leftSimdQuartf.y,leftSimdQuartf.z) and applied the above coords on leftEyeNode like self.leftEyeNode.eulerAngles = leftEyeEuler but seems like orientation is not matching with original one. Am I missing something – Chaitu Feb 20 '20 at 06:31
  • You don't need to try to apply `eulerAngles` in this case. A node's `orientation`, `rotation`, and `eulerAngles` properties are all interconnected, and changing one changes the others. You should only need to set the orientation. If that still doesn't look right, it's hard to say what might be wrong in relation to the rest of your model without more information. – jfriesenhahn Feb 21 '20 at 15:27
  • ok. thank you for suggestion. that means , setting the orientation for scnnode requires SCNQuaternion. So I have tried to converting the faceAnchor.leftEyeTransform into SCNQuaternion. and applied on the model . but seems like I am missing something not able to understand. Eye ball is moving towards x axis(up & down) but not on y axis (left to right).. – Chaitu Feb 21 '20 at 16:28
  • I can't really tell you what's going on without seeing more of your code. What do you mean the eye is moving? Are you doing an animation? Or do you just mean it is orienting, but not correctly? Is your `eyeTransform` a local/relative transform, or is it the `worldTransform`? What transform component are you applying it to on your model? Is it relative? If so, to what? – jfriesenhahn Feb 21 '20 at 23:11