0

I would like to read a movie file frame by frame, run some image processing algorithms on each frame, and display the resultant frames sequentially in an iOS app. There are mechanisms for doing this with the live camera feed, such as using an AVCaptureSession, and I would like to do something similar with movies already saved to disk.

I am trying to do this with an AVAssetReader and AVAssetReaderTrackOutput, however in this document, it clearly states that "AVAssetReader is not intended for use with real-time sources, and its performance is not guaranteed for real-time operations".

So my question is: what is the right way to read movie frames in real-time?

Edit:

To further clarify what I mean by "real-time", consider the fact that it possible to capture frames from the camera feed, run some computer vision algorithms on each frame (i.e. object detection, filtering, etc), and display the filtered frames at a reasonable frame rate (30 - 60 FPS). I would like to do the exact same thing, except that the input source is a video already saved on disk.

I don't want to read the entire movie, process the whole thing, and display the result only once the entire processing pipeline is finished (I would consider that non-real time). I want the processing to be done frame by frame, and in order to do that the file has to be read in frame by frame in real time.

ha314
  • 33
  • 5
  • 1
    Makes no sense. On disk and real time are opposites. – matt Jun 30 '21 at 04:00
  • Also how does this relate to your https://stackoverflow.com/questions/68169783/app-crashes-with-xpc-error-when-reading-video-file-with-avassetreader – matt Jun 30 '21 at 04:07
  • It doesn't make a lot of sense, try to explain better what you'd like to obtain and why you need "real time". AVFoundation is really a rich framework I'm pretty sure that you''ll find what you are looking for. – Andrea Jun 30 '21 at 04:30
  • @matt please see the edit regarding what I mean by "real time". As far as relation to the other question, that question is my attempt to implement this with AVAssetReader. My reason for asking this question is that if it turns out that AVAssetReader is not suitable for this purpose to being with, I would like to know a better alternative. – ha314 Jun 30 '21 at 05:07

1 Answers1

0

In order to playback and process a video in real-time you can use the AVPlayer class. The simplest way to live-process video frames is through a custom video composition on the AVPlayerItem.

You might want to check out this sample project from Apple where they highlight HDR parts in a video using Core Image filters. It shows the whole setup required for real-time processing and playback.

Frank Rupprecht
  • 9,191
  • 31
  • 56