2

I am trying to read frames of a video using the method in the accepted answer in this question: Accesing Individual Frames using AV Player.

I am then trying to display these frames sequentially on a MetalKit MKTView (I tried on a regular UIImage view, and the video doesn't render at all). The problem is that this only works for a few seconds (about 400 frames of the video), after which the app crashes and the video stops playing.

Here is the code in the view controller:

class MetalRenderViewController: UIViewController {
    var metalView: MetalRenderView!
    
    var videoFileReader: VideoFileReader?
    var videoUrl: URL!
    override func viewDidLoad() {
        super.viewDidLoad()
        metalView = MetalRenderView(frame: view.bounds, device: MTLCreateSystemDefaultDevice())
        view.addSubview(metalView)
        videoFileReader = VideoFileReader(videoUrl: videoUrl) { (image) in
            guard let image = image else {return}
            self.metalView.image = image

        }
        videoFileReader?.beginReadingAsset()
    }
    
}

And here is the relevant code in the videoFileReader:

guard let track = asset.tracks(withMediaType: .video).first else {return}
let reader = try! AVAssetReader(asset: asset)
let output = AVAssetReaderTrackOutput(track: track, outputSettings: settings)
reader.add(output)
reader.startReading()

while let sampleBuffer = output.copyNextSampleBuffer() {
            guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
            DispatchQueue.main.async {
                let image = CIImage(cvImageBuffer: imageBuffer)
                self.callback(image)
                
            }
        }

I suspect that the issue may be related to high memory utilization, since in the debugger the memory usage spikes to about 1.5 GB, and then falls down after the crash, although I don't see any memory warnings.

Here is the error log:

2021-06-28 16:13:22.491431-0400 WorkoutApp[10844:2837856] [xpc] XPC error talking to pkd: Connection interrupted
2021-06-28 16:13:22.773817-0400 WorkoutApp[10844:2837835] [ServicesDaemonManager] interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke
2021-06-28 16:13:28.057141-0400 WorkoutApp[10844:2837857] [lifecycle] [u 5485F2DA-5C8A-40DD-91A5-F248E652AA58:m (null)] [com.apple.mobileslideshow.photo-picker(1.0)] Connection to plugin interrupted while in use.
2021-06-28 16:13:28.075489-0400 WorkoutApp[10844:2838140] [lifecycle] [u 5485F2DA-5C8A-40DD-91A5-F248E652AA58:m (null)] [com.apple.mobileslideshow.photo-picker(1.0)] Connection to plugin invalidated while in use.

If someone could help me either reduce the memory utilization, or suggest a better way of doing this entirely, it would be greatly appreciated.

Edit

Updating title to be more specific about the error, removing Metal rendering classes which were a red herring.

ha314
  • 33
  • 5

1 Answers1

0

I was able to isolate this issue to the while loop in the VideoFileReader class by confirming that the issue still occurs after removing the ViewController and Metal rendering classes. I was also able to confirm that the crash was indeed due to high memory utilization by counting the number of frames read before the crash and comparing with the memory usage growth in the debugger.

What finally fixed the issue was wrapping the while-loop body in an autoreleasepool block. It seems that image buffers were being created and not released, and this was the cause of the memory spike and subsequent crash.

ha314
  • 33
  • 5