ARSCNView freezes after adding 12 ARFrame to Array (iOS/Swift)

Question

I have an ARSCNView in the ViewController.swift and I want to save the ARFrames to a pre-allocated array in

func session(_ session: ARSession, didUpdate frame: ARFrame)

However, after processing about 11-13 ARFrames the whole ARSCNView will freeze by using

self.ARFrames.append(frame)

What makes it strange is that func session(_ session: ARSession, didFailWithError error: Error) is not calling during the process, nor any other errors were reported, the app doesn't crash and every other user control works fine, only ARSCNView freeze and didpUdate event won't be called. Similar to ARSCNView freezes when adding 14 ARAnchor subclass objects with strong reference but the pages there doesn't have a solution. Also after app goes to the background and returns back, sessionWasInterrupted(:) and sessionInterruptionEnded(:) being called, even though scene view was freezed before. Is this a bug of iOS 11?

Here's the full code I am using in my app.

import UIKit

class ViewController: UIViewController,ARSCNViewDelegate,ARSessionDelegate {
    @IBOutlet var sceneView: ARSCNView!
    let configuration = ARFaceTrackingConfiguration()
    var ARFrames = [ARFrame]()
    var imgCount = 0

    override func viewDidLoad() {
        super.viewDidLoad()
        ARFrames.reserveCapacity(300)
        sceneView.delegate = self
        sceneView.session.delegate = self
        sceneView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
    }

    func session(_ session: ARSession, didUpdate frame: ARFrame) {
        if (frame.capturedDepthData == nil || self.imgCount >= 300){
            return
        }
        DispatchQueue.global().async {
            self.ARFrames.append(frame)
            self.imgCount += 1
        }
    }
}

score 7 · Accepted Answer · edited Jun 20 '20 at 09:12

7

Each ARFrame contains a video frame direct from the camera capture system (in its capturedImage property).

Each frame vended by the capture system comes from a fixed-size pool of memory, which the capture system reuses as the session continues. As noted in the capture docs:

If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBuffer objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

By adding all the ARFrames you get to an array, you’re claiming ownership of (i.e. retaining) their pixel buffers and eventually starving the capture system of memory to write new frames in. And ARKit needs a continuous stream of video, so your AR session gives up.

The solution? Don’t hold onto all those frames. Copy only whatever information you need from each frame into your own data structures.

edited Jun 20 '20 at 09:12

Community

1
1

answered Jul 18 '18 at 05:32

rickster

124,678
26
272
326

Thanks a lot, I was really not aware about this. – 崔振宇 Jul 18 '18 at 06:30
I thought there was something in the ARKit docs about this, but couldn’t find it. But the more folks [file bugs](http://bugreport.apple.com) on the docs, the better, right? – rickster Jul 18 '18 at 06:46
hi @rickster I am doing the same thing, appending frames to an array and encuntered the 12 frames limit. I am not sure about the meaning of copying and releasing the sample buffer? Do you mean creating a new CVPixelBuffer variable and store the ARFrame.captureimage in it? Thank you for your time! – mikey Apr 08 '19 at 07:00
@mikey No, you need to actually copy the pixel buffer data. See [this question](https://stackoverflow.com/q/37418517). – rickster Apr 08 '19 at 19:54
thank you @rickster I end up using an extension like https://gist.github.com/humblehacker/a55db40791605c4e40411f70bcd13d13, will maybe post an answer once I am done – mikey Apr 09 '19 at 01:00

ARSCNView freezes after adding 12 ARFrame to Array (iOS/Swift)

1 Answers1