RealityKit – Visualizing Grid on detected planes

Question

I want to draw the mesh point on the detected plane as shown in the ARCore video link and I don't know how to achieve it.

Can you help me out to achieve this? Thanks in advance.

Answer 1

Classic grid visualization in ARKit|RealityKit

In this post, I want to show you how to enable the "classic" visualization of the plane detection process with ARKit/RealityKit frameworks. But in this case, I don't promise you to visualize a grid in such a way that is visually pleasing, as it's implemented in Google ARCore.

However, in order to visualize the process of detecting horizontal and vertical planes similar to the behavior you can see in ARCore, you should use the scene reconstruction methodology, i.e. using a LiDAR scanner. But that's a different story, because you'll have to use Metal to implement a procedural texture with a soft mask for the front edge.

Here's a code:

import ARKit
import RealityKit

class Grid: Entity, HasModel, HasAnchoring {        
    var planeAnchor: ARPlaneAnchor
    var planeGeometry: MeshResource!
    
    init(planeAnchor: ARPlaneAnchor) {
        self.planeAnchor = planeAnchor
        super.init()
        self.didSetup()
    }
        
    fileprivate func didSetup() {            
        self.planeGeometry = .generatePlane(width: planeAnchor.extent.x,
                                            depth: planeAnchor.extent.z)
        var material = UnlitMaterial()
        material.color = .init(tint: .white.withAlphaComponent(0.999),
                            texture: .init(try! .load(named: "grid.png")))
        let model = ModelEntity(mesh: planeGeometry, materials: [material])
        model.position = [planeAnchor.center.x, 0, planeAnchor.center.z]
        self.addChild(model)
    }
    
    fileprivate func didUpdate(anchor: ARPlaneAnchor) {            
        self.planeGeometry = .generatePlane(width: anchor.extent.x,
                                            depth: anchor.extent.z)
        let pose: SIMD3<Float> = [anchor.center.x, 0, anchor.center.z]
        let model = self.children[0] as! ModelEntity
        model.position = pose
    }        
    required init() { fatalError("Hasn't been implemented yet") }
}

ViewController.swift

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    var grids = [Grid]()

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        arView.session.delegate = self
        
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal, .vertical]
        arView.session.run(config)
    }
}

ARSessionDelegate

extension ViewController: ARSessionDelegate {
    
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
        
        guard let planeAnchor = anchors.first as? ARPlaneAnchor else { return }
        let grid = Grid(planeAnchor: planeAnchor)
        grid.transform.matrix = planeAnchor.transform
        self.arView.scene.anchors.append(grid)
        self.grids.append(grid)
    }
    
    func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
        
        guard let planeAnchor = anchors[0] as? ARPlaneAnchor else { return }
        let grid: Grid? = grids.filter { grd in
            grd.planeAnchor.identifier == planeAnchor.identifier }[0]
        guard let updatedGrid: Grid = grid else { return }
        updatedGrid.transform.matrix = planeAnchor.transform
        updatedGrid.didUpdate(anchor: planeAnchor)
    }
}

Only coplanar detected planes may be updated.

P. S.

If you're interested in visualizing the Canonical Face Mask in RealityKit, take a look at this post.