ARKit – Replicating robot character in Motion Capture RealityKit

Question

I'm trying to make a 3d model like robot provided by Apple in Motion Capture example (shown at WWDC 2019) which can mimic me in motion capture ARKit 3.0 by replacing robot character given by Apple.

Desired Solution:

• Is there any special software which Apple used to create robot.usdz file? If yes, then please provide details for it?

• How can we convert formats like .glb/.gltf/.obj/.dae file to .usdz using Apple’s Python based tool without affecting it’s scene graph?

• How can we edit the scene graph of a .usdz file in Xcode and successfully save the changes in a .usdz file?

Answer 1

Working ARKit's MoCap solution

• Import or create a character in Autodesk Maya 2024
• Download Apple's model biped skeleton
• Appropriately rig a Apple's skeleton for a Motion Capture
• Bind skeleton to a mesh using Skin–Bind Skin menu
• Select skeleton and character mesh in Outliner
• Export a model as USD file format with baked animation
• Optionally download USDZ Python Tools and FBX Python SDK

Here's my post where you'll find instructions how to create a .zshrc file in macOS.

Load MoCap'ed model in RealityKit:

import RealityKit
import ARKit

class ViewController: UIViewController, ARSessionDelegate {

    @IBOutlet var arView: ARView!
    
    var character: Entity?
    let characterAnchor = AnchorEntity()
    
    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        arView.session.delegate = self

        guard ARBodyTrackingConfiguration.isSupported
        else { fatalError("MoCap is available on A12 & later") }
        
        let config = ARBodyTrackingConfiguration()
        arView.session.run(config)
        arView.scene.addAnchor(characterAnchor)
        
        character = try? Entity.load(named: "character")
    }
    
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        
        for anchor in anchors {
            
            guard let bodyAnchor = anchor as? ARBodyAnchor
            else { continue }

            let bodyPosition = simd_make_float3(bodyAnchor.transform.columns.3)
            characterAnchor.position = bodyPosition
            characterAnchor.orientation = Transform(matrix: bodyAnchor.transform).rotation

            if let character = character, character.parent == nil {

                characterAnchor.addChild(character)
                characterAnchor.scale = [0.02, 0.02, 0.02]
            }
        }
    }
}