1

I am trying to create an app where I can use the depth functionalities of RealityKit but the AR drawing capabilities from SceneKit. What I would like to do, is recognize an object and place a 3d model over it (which works already).

When that is completed I would like the user to be able to draw on top of that 3d model (which works fine with SceneKit, but makes the 3d model jitter). I found SCNLine to do the drawing, but since it uses SceneKit I can not use it in the ARView of RealityKit.

I have seen this already, but it does not cover fully what I would like.

Is it possible to use both?

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
Geart Otten
  • 241
  • 4
  • 11

1 Answers1

2

SceneKit and RealityKit are incompatible due to a complete dissimilarity – difference in scenes' hierarchy, difference in renderer and physics engines, difference in component content. What's stopping you from using SceneKit + ARKit (ARSCNView class)?

ARKit 6.0 has a built-in Depth API (the same API is available in RealityKit) that uses a LiDAR scanner to more accurately determine distances in a surrounding environment, allowing us to use plane detection, raycasting and object occlusion more efficiently.

For that, use sceneReconstruction instance property and ARMeshAnchors.

import ARKit
import SceneKit

class ViewController: UIViewController {

    @IBOutlet var sceneView: ARSCNView!

    override func viewDidLoad() {
        super.viewDidLoad()

        sceneView.scene = SCNScene()
        sceneView.delegate = self

        let config = ARWorldTrackingConfiguration()
        config.sceneReconstruction = .mesh
        config.planeDetection = .horizontal
        sceneView.session.run(config)
    }
}

Delegate's method:

extension ViewController: ARSCNViewDelegate {

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, 
                                                 for anchor: ARAnchor) {

        guard let meshAnchor = anchor as? ARMeshAnchor else { return }
        let meshGeo = meshAnchor.geometry

        // logic ...

        node.addChildNode(someModel)
    }
}

P. S.

This post will be helpful for you.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
  • I was unsure that SceneKit + ARKit used the same depth-technology as RealityKit. Which I'll be trying to implement now, hopefully the shaking will be gone. – Geart Otten Jul 22 '22 at 07:13
  • 1
    Shaking is the result of poor tracking, not the result of calculating per-pixel depth. Read about `session.currentFrame?.sceneDepth`– `.sceneDepth` object's type is ARDepthData (ARKit's class). RealityKit's session is based on ARKit. – Andy Jazz Jul 22 '22 at 07:24