CMSampleBuffer automatically rotate every time

Question

Here I try to select video from photo library and read it frame by frame as a samplebuffer so that I can crop or rotate later.but the problem is CMSampleBuffer by default rotated.The variables are I use for initialization is

 var asset:AVAsset!     //load asset from url
 var assetReader:AVAssetReader!   
 var assetVideoOutput: AVAssetReaderTrackOutput!  // add assetreader for video
 var assetAudioOutput: AVAssetReaderTrackOutput!  // add assetreader for audio
 var readQueue: DispatchQueue!

the settings of previous variables looks like this.

func resetRendering() {

    do{
        assetReader = try! AVAssetReader(asset: asset)
    }catch {
        print(error)
    }

    var tracks = asset.tracks(withMediaType: .video)
    var track : AVAssetTrack!
    if tracks.count > 0 {
        track = tracks[0]
    }

    let decompressionvideoSettings:[String:Any] = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA]

    assetVideoOutput = AVAssetReaderTrackOutput(track: track, outputSettings: decompressionvideoSettings)

    tracks = asset.tracks(withMediaType: .audio)
    if tracks.count > 0 {
        track = tracks[0]
    }

    let audioReadSettings = [AVFormatIDKey: kAudioFormatLinearPCM]
    assetAudioOutput = AVAssetReaderTrackOutput(track: track, outputSettings: audioReadSettings)

    if (assetAudioOutput.responds(to: #selector(getter: AVAssetReaderOutput.alwaysCopiesSampleData))) {
        assetAudioOutput.alwaysCopiesSampleData = false
    }
    if assetReader.canAdd(assetAudioOutput) {
        assetReader.add(assetAudioOutput)
    }

    if (assetVideoOutput.responds(to: #selector(getter: AVAssetReaderOutput.alwaysCopiesSampleData))) {
        assetVideoOutput.alwaysCopiesSampleData = false
    }
    if assetReader.canAdd(assetVideoOutput) {
        assetReader.add(assetVideoOutput)
    }

}

now when I try to read video frame by frame and convert into image it automatically rotate frame in 90 degrees.the conversion extention from samplebuffer to uiimage looks like this

extension CMSampleBuffer {
    var uiImage: UIImage? {
        guard let imageBuffer = CMSampleBufferGetImageBuffer(self) else { return nil }

        CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags(rawValue: 0))
        let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
        let bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer)
        let width = CVPixelBufferGetWidth(imageBuffer)
        let height = CVPixelBufferGetHeight(imageBuffer)
        let colorSpace = CGColorSpaceCreateDeviceRGB()
        let bitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.noneSkipFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)
        guard let context = CGContext(data: baseAddress,
                                      width: width,
                                      height: height,
                                      bitsPerComponent: 8,
                                      bytesPerRow: bytesPerRow,
                                      space: colorSpace,
                                      bitmapInfo: bitmapInfo.rawValue) else { return nil }
        guard let cgImage = context.makeImage() else { return nil }

        CVPixelBufferUnlockBaseAddress(imageBuffer,CVPixelBufferLockFlags(rawValue: 0));

        return UIImage(cgImage: cgImage)
    }
}

resulted photo is

and actual photo is this one

score 0 · Answer 1 · edited Feb 05 '19 at 11:12

Update

CGImage doesn't hold orientation property. So you have to manually set when you convert to UIImage. We get orientation from video track and set it to UIImage.

Below answer's right/left or up/down might be opposite. Pls try and check.

//pls get orientation property outside of image processing to avoid redundancy
let videoTrack = videoAsset.tracks(withMediaType: AVMediaType.video).first!
let videoSize = videoTrack.naturalSize
let transform = videoTrack.preferredTransform

var orientation = UIImage.Orientation.right
switch (transform.tx, transform.ty) {
case (0, 0): orientation = UIImage.Orientation.up
case (videoSize.width, videoSize.height): UIImage.Orientation.right
case (0, videoSize.width): UIImage.Orientation.left
default: orientation = UIImage.Orientation.down
}


// then set orientation in image processing such as captureOutput
let uiImage = UIImage.init(cgImage: cgImage, scale: 1.0, orientation: orientation)

Hope this helps.

Some video it gives proper orientation but some are not.So if I rotate every one then it may not help me for some video. — MD. Rejaul Hasan, Jan 21 '19 at 05:18
Catch the orientation for each video and set proper one. As cgImage forgets orientation data, we get it from video track and set at UIImage.init(: : :). Pls refer to cleverbit's answer on this post for how you get orientation data. https://stackoverflow.com/questions/4627940/how-to-detect-if-a-video-file-was-recorded-in-portrait-orientation-or-landscape — Giraff Wombat, Jan 21 '19 at 22:51

CMSampleBuffer automatically rotate every time

1 Answers1