1

I'm trying to reverse engineer the 3d Scanner App using RealityKit and am having real trouble getting just a basic model working with all gestures. When I run the code below, I get a cube with scale and rotation (about the y axis only), but no translation interaction. I'm trying to figure out how to get rotation about an arbitray axis as well as translation, like in the 3d Scanner App above. I'm relatively new to iOS and read one should use RealityKit as Apple isn't really supporting SceneKit anymore, but am now wondering if SceneKit would be the way to go, as RealityKit is still young. Or if anyone knows of an extension to RealityKit ModelEntity objects to give them better interaction capabilities.

I've got my app taking a scan with the LiDAR sensor and saving it to disk as a .usda mesh, per this tutorial, but when I load the mesh as a ModelEntity and attach gestures to it, I don't get any interaction at all.

The below example code recreates the limited gestures for a box ModelEntity, and I have some commented lines showing where I would load my .usda model from disk, but again while it will render, it gets no interaction with gestures.

Any help appreciated!

// ViewController.swift
import UIKit
import RealityKit

class ViewController: UIViewController {

    var arView: ARView!
    
    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view.
        
        arView = ARView(frame: view.frame, cameraMode: .nonAR, automaticallyConfigureSession: false)
        view.addSubview(arView)
        
        // create pointlight
        let pointLight = PointLight()
        pointLight.light.intensity = 10000
        // create light anchor
        let lightAnchor = AnchorEntity(world: [0, 0, 0])
        lightAnchor.addChild(pointLight)
        arView.scene.addAnchor(lightAnchor)
        
//      eventually want to load my model from disk and give it gestures.
//        guard let scanEntity = try? Entity.loadModel(contentsOf: urlOBJ) else {
//            print("couldn't load scan in this format")
//            return
//        }

        // entity to add gestures to
        let cubeMaterial = SimpleMaterial(color: .blue, isMetallic: true)
        let myEntity = ModelEntity(mesh: .generateBox(width: 0.1, height: 0.2, depth: 0.3, cornerRadius: 0.01, splitFaces: false), materials: [cubeMaterial])
        
        myEntity.generateCollisionShapes(recursive: false)
        
        let myAnchor = AnchorEntity(world: .zero)
        myAnchor.addChild(myEntity)
        
        // add collision and interaction
        let scanEntityBounds = myEntity.visualBounds(relativeTo: myAnchor)
        myEntity.collision = CollisionComponent(shapes: [.generateBox(size: scanEntityBounds.extents).offsetBy(translation: scanEntityBounds.center)])
        
        arView.installGestures(for: myEntity).forEach {
            gestureRecognizer in
            gestureRecognizer.addTarget(self, action: #selector(handleGesture(_:)))
        }
        arView.scene.addAnchor(myAnchor)
        
        // without this, get no gestures at all
        let camera = PerspectiveCamera()
        let cameraAnchor = AnchorEntity(world: [0, 0, 0.2])
        cameraAnchor.addChild(camera)
        arView.scene.addAnchor(cameraAnchor)
    }
    
    @objc private func handleGesture(_ recognizer: UIGestureRecognizer) {
        if recognizer is EntityTranslationGestureRecognizer {
            print("translation!")
        } else if recognizer is EntityScaleGestureRecognizer {
            print("scale!")
        } else if recognizer is EntityRotationGestureRecognizer {
            print("rotation!")
        }
    }

}
lewis
  • 111
  • 1
  • 4

2 Answers2

0

To extend ModelEntity's gesture interaction capabilities setup your own 2D gestures. There are 8 screen gestures in UIKit, and in SwiftUI you have 5 principal gestures and additionally Sequence, Simultaneous and Exclusive variations.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
0

Form what I have understood, that the gestures are working for the box but not for your .usdz file/model. If this is the case, then the issue is because the model does not have a collision mesh(HasCollsion). If you are using reality composer to edit your models, you could do the following:

  1. click on the model
  2. under the Physics dropdown, click Participate
  3. under collision shape select automatic

Overalls, make sure that the model has collision and you cast within the code that it has collision

let myEntity = try? Entity.loadModel(named: "fileName") as! HasCollision
I_Wish_I_was_That_Good
  • 11
  • 1