Accesing Individual Frames using AV Player

Question

In a recent project, I have to access all frames of my video individually using AV Foundation. Also, if possible to acess them randomly (like an array)

I tried to research the question but I didn't get anything useful.

Note: Is there any useful documentation to get familiar with the AV Foundation ?

I can answer you for any question. But you have to explain more your question/Problem.. — JAck, Sep 19 '16 at 11:15
The problem is that I don't know how to access individual frames of my video. Because I need to apply some magic algorithm in each frame by itself. — user2232305, Sep 19 '16 at 11:28
yes, I got it now. You want to access frames of videos. right?? — JAck, Sep 19 '16 at 12:04

score 12 · Accepted Answer · edited May 23 '17 at 10:34

12

You can enumerate the frames of your video serially using AVAssetReader, like this:

let asset = AVAsset(URL: inputUrl)
let reader = try! AVAssetReader(asset: asset)

let videoTrack = asset.tracksWithMediaType(AVMediaTypeVideo)[0]

// read video frames as BGRA
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings:[String(kCVPixelBufferPixelFormatTypeKey): NSNumber(unsignedInt: kCVPixelFormatType_32BGRA)])

reader.addOutput(trackReaderOutput)
reader.startReading()

while let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
    print("sample at time \(CMSampleBufferGetPresentationTimeStamp(sampleBuffer))")
    if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
        // process each CVPixelBufferRef here
        // see CVPixelBufferGetWidth, CVPixelBufferLockBaseAddress, CVPixelBufferGetBaseAddress, etc
    }
}

Random access is more complicated. You could use an AVPlayer + AVPlayerItemVideoOutput to get frames from any time t, using copyPixelBufferForItemTime, as described in this answer, but the subtlety lies in how you choose that t.

If you want to sample the video at uniform intervals, then that's easy, but if you want to land on the same frames/presentation time stamps that the serial AVAssetReader code sees, then you will probably have to preprocess the file with AVAssetReader, to build a frame number -> presentation timestamp map. This can be fast if you skip decoding by using nil output settings in AVAssetReaderTrackOutput.

edited May 23 '17 at 10:34

Community

1
1

answered Sep 19 '16 at 12:52

Rhythmic Fistman

34,352
5
87
159

Does this give all frames of my video ? – user2232305 Sep 19 '16 at 13:33
The `AVAssetReader` method does. If you know all the frame timestamps, the "random access" method should too. – Rhythmic Fistman Sep 19 '16 at 13:34
Thanks, that's very helpful. One question(s), can you provide with a documentation about CMSampleBufferGetImageBuffer because I still don't get it. Also, is there a way to save locally them so that I check what is happenning ? – user2232305 Sep 19 '16 at 13:43
There's some info in the `CMSampleBuffer` header file. It returns a `CVPixelBufferRef` from which you can get width, height and a pointer to pixels. See http://stackoverflow.com/q/8072208/22147 for `CVPixelBuffer`->`UIImage`. – Rhythmic Fistman Sep 19 '16 at 13:53
Thanks. If I have any problem I will write to you. – user2232305 Sep 19 '16 at 13:58
Hi Fistman, I have another question please. For the AVAssetReader is meant for offline (not real time) what about getting them in real time ? – user2232305 Sep 21 '16 at 12:40
What do you mean by realtime? – Rhythmic Fistman Sep 21 '16 at 13:29
Assume I have a video on a server and I can play it on a local player. What I need is to acess its frames on real time (while it is playing for example) and apply some magic on it. – user2232305 Sep 21 '16 at 13:52
AVAssetReader only does local files, so you need AVPlayer + videooutput. – Rhythmic Fistman Sep 21 '16 at 13:53
Can you explain how to do so please ? And if you have any documentation about it please let me know. – user2232305 Sep 21 '16 at 14:18
The explanation is in the answer? Maybe start a new question with these new details. – Rhythmic Fistman Sep 21 '16 at 14:22

Mudith Chathuranga Silva · Answer 2 · 2023-06-26T08:38:08.040

If you need to get the current frame while playing the video you could add an observer for the player and could get the current frame like this:

var player: AVPlayer?
var playerController = AVPlayerViewController()
var videoFPS: Int = 0
var currentFrame: Int = 0
var totalFrames: Int?

func configureVideo() {
    guard let videoURL = "" else { return }
    player = AVPlayer(url: videoURL)
    playerController.player = player
    guard player?.currentItem?.asset != nil else {
        return
    }
    let asset = self.player?.currentItem?.asset
    let tracks = asset!.tracks(withMediaType: .video)
    let fps = tracks.first?.nominalFrameRate
    
    self.videoFPS = lround(Double(fps!))
    self.getVideoData()
}
  
func getVideoData() {
    self.player?.addPeriodicTimeObserver(forInterval: CMTimeMake(value: 1,timescale: Int32(self.videoFPS)), queue: DispatchQueue.main) {[weak self] (progressTime) in

        if let duration = self!.player?.currentItem?.duration {

            let durationSeconds = CMTimeGetSeconds(duration)
            let seconds = CMTimeGetSeconds(progressTime)
            if self!.totalFrames == nil {
                self!.totalFrames = lround(Double(self!.videoFPS) * durationSeconds)
            }

            DispatchQueue.main.async {
                if self!.totalFrames != self!.currentFrame {
                    self!.currentFrame = lround(seconds*Double(self!.videoFPS))
                } else {
                    print("Video has ended!!!!!!!!")
                }
                print(self!.currentFrame)
            }
        }
    }
}

After getting the current frame number, you could use AVAssetImageGenerator to retrieve the specific image.

Accesing Individual Frames using AV Player

2 Answers2

Linked