Show image on SCNPlane in Scene kit & AR kit in swift iOS

Question

I followed this raywenderlich tutorial to detect image using AR and Scene Kit. In this example they showed displaying video on top of detected image.

https://www.raywenderlich.com/6957-building-a-museum-app-with-arkit-2

I want to display image on top of detected image. Please help me how can I add childNode to SCNNode so that I can display image on top of it.

Here is the code which I tried it's not working showing a blank white screen.

                            DispatchQueue.main.async {

                        let size = imageAnchor.referenceImage.physicalSize

                        let imageView = UIImageView(image: UIImage(named: "BG.png")!)

                        let imgMaterial = SCNMaterial()
                        imgMaterial.diffuse.contents = imageView

                        let imgPlane = SCNPlane(width: size.width, height: size.height)
                        imgPlane.materials = [imgMaterial]

                        let imgNode = SCNNode(geometry: imgPlane)
                        imgNode.eulerAngles.x = -.pi / 2


                        node.addChildNode(imgNode)
                         node.opacity = 1

                        }

Please help to show image on top of detected image. Thanks in advance.

I'm able to show image on top of detected image but it takes lot of memory what is wrong in that code.

And same as image view I'm displaying gif play on top of detected image with the below code.

                                let size = imageAnchor.referenceImage.physicalSize

                            let imgNameFromDoc = person["actionToTake"] as! String

                            let documentsPathURL = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first

                            let imgpath = documentsPathURL?.appendingPathComponent("/Packages/\(self.selectedDatasetName)/3DScenes/\(imgNameFromDoc)")

                            let imageData = try! Data(contentsOf: URL(fileURLWithPath: imgpath!.path))


                            let imageURL = UIImage.gifImageWithData(imageData)


                            self.gifImgView = UIImageView(image: imageURL)

                            let imgMaterial = SCNMaterial()
                            imgMaterial.diffuse.contents = self.gifImgView

                            let imgPlane = SCNPlane(width: CGFloat(scaleActionSize(wid: Float(size.width), hei:Float(size.height))[0]), height: CGFloat(scaleActionSize(wid: Float(size.width), hei:Float(size.height))[1]))

                            imgPlane.materials = [imgMaterial]

                            let imgNode = SCNNode(geometry: imgPlane)
                            imgNode.eulerAngles.x = -.pi / 2


                            node.addChildNode(imgNode)
                            node.opacity = 1

With the help of image extension I'm showing GIF on image view on top of detected image. https://github.com/kiritmodi2702/GIF-Swift/blob/master/GIF-Swift/iOSDevCenters%2BGIF.swift

How to load GIF image in Swift?

I dragged iOSDevCenters+GIF.swift class inside app and displayed gif.

When I use this gif image viewto display gif and image view to display image it takes more than 800 MB memory in Xcode running with iPad.

can somebody help me to find any memory leaks(tried with instruments not able to fix) or anything wrong in this code.

Answer 1

I guess you have image tracking working.

The next method is called when a new anchor is added. In this case when your image tracked is detected. So the parameter node is the image tracked and you can add a child to that image as in the tutorial. So you can try the next (use ARSCNViewDelegate):

    func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {

    DispatchQueue.main.async {
        let imageView = UIImageView(image: UIImage())

        let imgMaterial = SCNMaterial()
        imgMaterial.diffuse.contents = imageView

        let imgPlane = SCNPlane(width: 0.1, height: 0.1)
        imgPlane.materials = [imgMaterial]

        let imgNode = SCNNode(geometry: imgPlane)
        imgNode.eulerAngles.x = -.pi / 2


        node.addChildNode(imgNode)
        node.opacity = 1

    }

}

The imgNode is added to the node and the node is already added to the scene. So what happen if the node is not added to the scene? You have to add it by your own. So in case you are not using the renderer method, an example of it is the next one:

self.imgNode?.position = SCNVector3(0, 0, 0)
sceneView.scene.rootNode.addChildNode(self.imgNode)

Answer 2

Displaying UIKit views as material property contents is not supported. Instead of a UIImageView you should directly use the UIImage or even better just a String or URL that points to the image on disk. This way the image won't be loaded twice in memory (in SceneKit and UIKit).