Getting a RealityKit plane to hold still when the camera moves

Question

Follow up to the answer to Keeping model anchored in place when a tracking image is hidden

Implementing the answer with a plane instead of a sphere looks like this:

import ARKit
import RealityKit

class ViewController : UIViewController {

  @IBOutlet var arView: ARView!
  var anchorEntity = AnchorEntity(.image(group: "AR", name: "Image"))
  var planeEntity = ModelEntity()

  override func viewDidLoad() {
    super.viewDidLoad()

    arView.session.delegate = self

    // size of image referenced above
    let width = Float(0.2667)
    let height = Float(0.15001875)
    let mesh: MeshResource = .generatePlane(width: width, depth: height)
    let material = SimpleMaterial(color: .systemPink, isMetallic: false)
    self.planeEntity = ModelEntity(mesh: mesh, materials: [material])

    self.anchorEntity.addChild(planeEntity)
    arView.scene.anchors.append(self.anchorEntity)
  }
}

extension ViewController : ARSessionDelegate {

  func session(_ session: ARSession, didUpdate frame: ARFrame) {

    if anchorEntity.isActive {
      anchorEntity.reanchor(.world(transform: planeEntity.transform.matrix),
                      preservingWorldTransform: true)
    }
  }
}

If you run that with your own image, you'll see that the plane is not firmly anchored in place at all. It doesn't move around as much as it would without the call to reanchor, but it's still quite mobile. Is this the best I can do?

Second, in my actual application I want the plane to appear when the tracked image disappears, rather than when it appears, and then remain anchored in the spot where the image was last seen.

I've been tinkering with it and this is the best implementation I've been able to come up with. It's not completely still, but it moves around less than any other approach I've tried. Is there a better way? I've tried calling reanchor here too but it doesn't make any difference.

class ViewController: UIViewController {

  @IBOutlet var arView: ARView!
  var anchorEntity = AnchorEntity()
  var planeEntity = ModelEntity()

  override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
    arView.session.delegate = self

    guard let reference = ARReferenceImage.referenceImages(
      inGroupNamed: "AR",
      bundle: nil)
    else { return }

    let config = ARWorldTrackingConfiguration()
    config.detectionImages = reference
    config.maximumNumberOfTrackedImages = 1
    arView.session.run(config)

    let width = Float(0.2667)
    let height = Float(0.15001875)
    let mesh: MeshResource = .generatePlane(width: width, depth: height)
    let material = SimpleMaterial(color: .systemPink, isMetallic: false)
    self.planeEntity = ModelEntity(mesh: mesh, materials: [material])

    self.anchorEntity.addChild(planeEntity)
  }
}

extension ViewController: ARSessionDelegate {

  func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {

    guard let imageAnchor = anchors.first as? ARImageAnchor
    else { return }

    if imageAnchor.isTracked == false {
      arView.scene.anchors.append(self.anchorEntity)
      self.planeEntity.transform.matrix = imageAnchor.transform
      self.anchorEntity = AnchorEntity(anchor: imageAnchor)
    }
  }
}

score 0 · Answer 1 · answered Jun 10 '23 at 11:24

I can't answer the second part, but the first part is something that happened to me too. I used the same reanchor(.world(transform: planeEntity.transform.matrix), preservingWorldTransform: true) solution, and the reanchored object kept moving around. Eventually, I realized it's because of the original image anchor. The size of the image, as defined in the AR Resource Group under the Assets, is crucial for ARKit to determine the distance of the anchor from the camera; and when I inputted the wrong size, it was placed in a way that caused these movements.

See if it applies to you too, and more details here: https://developer.apple.com/documentation/arkit/arreferenceimage/2941027-physicalsize

Getting a RealityKit plane to hold still when the camera moves

1 Answers1