Detect current Keyframe interval in AVAsset

Question

I am working on an application that plays back video and allows the user to scrub forwards and backwards in the video. The scrubbing has to happen smoothly, so we always re-write the video with SDAVAssetExportSession with the video compression property AVVideoMaxKeyFrameIntervalKey:@1 so that each frame will be a keyframe and allow smooth reverse scrubbing. This works great and provides smooth playback. The application uses video from a variety of sources and can be recorded on android or iOS devices and even downloaded from the web and added to the application, so we end up with quite different encodings, some of which are already suited for scrubbing (each frame is a keyframe). Is there a way to detect the keyframe interval of a video file so I can avoid needless video processing? I have been through much of AVFoundation's docs and don't see an obvious way to get this information. Thanks for any help on this.

Answer 1

If you can quickly parse the file without decoding the images by creating an AVAssetReaderTrackOutput with nil outputSettings. The frame sample buffers you encounter have an attachment array containing a dictionary with useful information, include whether the frame depends on other frames, or whether other frames depend on it. I would interpret that former as indicating a keyframe, although it gives me some low number (4% keyframes in one file?). Anyway, the code:

let asset = AVAsset(url: inputUrl)
let reader = try! AVAssetReader(asset: asset)

let videoTrack = asset.tracks(withMediaType: AVMediaTypeVideo)[0]
let trackReaderOutput = AVAssetReaderTrackOutput(track: videoTrack, outputSettings: nil)

reader.add(trackReaderOutput)
reader.startReading()

var numFrames = 0
var keyFrames = 0

while true {
    if let sampleBuffer = trackReaderOutput.copyNextSampleBuffer() {
        // NB: not every sample buffer corresponds to a frame!
        if CMSampleBufferGetNumSamples(sampleBuffer) > 0 {
            numFrames += 1
            if let attachmentArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, false) as? NSArray {
                let attachment = attachmentArray[0] as! NSDictionary
                // print("attach on frame \(frame): \(attachment)")
                if let depends = attachment[kCMSampleAttachmentKey_DependsOnOthers] as? NSNumber {
                    if !depends.boolValue {
                        keyFrames += 1
                    }
                }
            }
        }
    } else {
        break
    }
}

print("\(keyFrames) on \(numFrames)")

N.B. This only works for local file assets.

p.s. you don't say how you're scrubbing or playing. An AVPlayerViewController and an AVPlayer?

Answer 2

Here is the Objective C version of the same answer. After implementing this and using it, Videos that should have all keyframes are returning about 96% keyframes from this code. I'm not sure why, so I am using that number as a determining factor even though I would like it to be more accurate. I am also only looking through the first 600 frames or the end of the video (whichever comes first) since I don't need to read through a whole 20 minute video to make this determination.

+ (BOOL)videoNeedsProcessingForSlomo:(NSURL*)fileUrl {
    BOOL needsProcessing = YES;
    AVAsset* anAsset = [AVAsset assetWithURL:fileUrl];
    NSError *error;
    AVAssetReader *assetReader = [AVAssetReader assetReaderWithAsset:anAsset error:&error];
    if (error) {
        DLog(@"Error:%@", error.localizedDescription);
        return YES;
    }

    AVAssetTrack *videoTrack = [[anAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];

    AVAssetReaderTrackOutput *trackOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:videoTrack outputSettings:nil];
    [assetReader addOutput:trackOutput];

    [assetReader startReading];

    float numFrames = 0;
    float keyFrames = 0;

    while (numFrames < 600) { // If the video is long - only parse through 20 seconds worth.
        CMSampleBufferRef sampleBuffer = [trackOutput copyNextSampleBuffer];
        if (sampleBuffer) {
            // NB: not every sample buffer corresponds to a frame!
            if (CMSampleBufferGetNumSamples(sampleBuffer) > 0) {
                numFrames += 1;

                NSArray *attachmentArray = ((NSArray*)CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, false));
                if (attachmentArray) {
                    NSDictionary *attachment = attachmentArray[0];
                    NSNumber *depends = attachment[(__bridge NSNumber*)kCMSampleAttachmentKey_DependsOnOthers];
                    if (depends) {
                        if (depends.boolValue) {
                            keyFrames += 1;
                        }
                    }
                }
            }
        }
        else {
            break;
        }
    }

    needsProcessing = keyFrames / numFrames < 0.95f; // If more than 95% of the frames are keyframes - don't decompress.

    return needsProcessing;
}

Answer 3

Using kCMSampleAttachmentKey_DependsOnOthers was giving me 0 key frames in some cases, when ffprobe would return key frames. To get the same number of key frames as ffprobe shows, I used:

if attachment[CMSampleBuffer.PerSampleAttachmentsDictionary.Key.notSync] == nil {
                            keyFrames += 1
                        }

In the CoreMedia header it says:

/// Boolean (absence of this key implies Sync)
public static let notSync: CMSampleBuffer.PerSampleAttachmentsDictionary.Key

for dependsOnOthers key it says:

/// `true` (e.g., non-I-frame), `false` (e.g. I-frame), or absent if
/// unknown
public static let dependsOnOthers: CMSampleBuffer.PerSampleAttachmentsDictionary.Key