Acessing Video Frames in Real Time iOS

Question

I have to access video from a server and I want to be able to acess its frames in real time so that I can do some image processing on them.

I know how to do it offline using the AVAssetReader but this solution does not work for real time processing.

Answer 1

You can access frames of a remote asset in realtime using AVPlayerItemVideoOutput (with thanks to objc.io).

p.s. always check your remote asset isn't being blocked by App Transport Security.

p.p.s AVAssetReader's support for remote assets is complicated.

import UIKit
import AVFoundation

class ViewController: UIViewController {        
    let player = AVPlayer(url: URL(string: "http://0.s3.envato.com/h264-video-previews/80fad324-9db4-11e3-bf3d-0050569255a8/490527.mp4")!)
    let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: [String(kCVPixelBufferPixelFormatTypeKey): NSNumber(value: kCVPixelFormatType_32BGRA)])

    override func viewDidLoad() {
        super.viewDidLoad()

        player.currentItem!.add(videoOutput)
        player.play()

        let displayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidRefresh(link:)))
        displayLink.add(to: RunLoop.main, forMode: RunLoopMode.commonModes)
    }

    func displayLinkDidRefresh(link: CADisplayLink) {
        let itemTime = videoOutput.itemTime(forHostTime: CACurrentMediaTime())

        if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
            if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: nil) {
                print("pixelBuffer \(pixelBuffer)") // yay, pixel buffer
                let image = CIImage(cvImageBuffer: pixelBuffer) // or maybe CIImage?
                print("CIImage \(image)")
            }
        }
    }
}