7

I've been struggling with several dimensions to the problem of controlling video orientation during and after capture on an iOS device. Thanks to previous answers and documentation from Apple I've been able to figure it out. However, now that I want to push some video to a web site, I'm running into particular problems. I've outlined this problem in particular in this question, and the proposed solution turns out to require orientation options to be set during video encoding.

That may be, but I have no clue how to go about doing this. The documentation around setting orientation is in respect to setting it correctly for display on the device, and I've implemented the advice found here. However, this advice does not address setting the orientation properly for non-Apple software, such as VLC or the Chrome browser.

Can anyone provide insight into how to set orientation properly on the device such that it displays correctly for all viewing software?

Community
  • 1
  • 1
Aaron Vegh
  • 5,217
  • 7
  • 48
  • 75
  • 2
    The actual data always has static orientation during capture. The orientation is stored in the `preferredTransform` value. So, I guess, you need to export video to rotate the data. I would look into `AVAssetExportSession` `AVMutableVideoComposition` `setTransform:atTime:`, this might help. – Davyd Geyl Nov 22 '12 at 00:02
  • I have a technical support incident request into Apple to help figure this out. But I'll take a look as you suggest. Would this mean a separate encode step after the video is recorded, I wonder? That might be computationally expensive... – Aaron Vegh Nov 22 '12 at 02:07
  • Yes, this would be an extra step. However, it may be not that expensive if export without changing the original encoding. Let me know if you find a better solution. – Davyd Geyl Nov 22 '12 at 10:41
  • I've successfully implemented your suggestion — I created an AVAssetExportSessionn using the AVMutableVideoComposition and setting a transform. It was a pretty hairy time, but I got it working this weekend. Thanks for the tip! – Aaron Vegh Nov 26 '12 at 03:17
  • Great! I am happy it worked for you. – Davyd Geyl Nov 26 '12 at 03:47
  • Hi Guys. Non of the options solve this for me (nowadays the problem is with FireFox and IE, chrome is good). Any leads? – Boaz Jul 12 '15 at 13:52

6 Answers6

10

Finally,based on the answers of @Aaron Vegh and @Prince, I figured out my resolution: //Converting video

+(void)convertMOVToMp4:(NSString *)movFilePath completion:(void (^)(NSString *mp4FilePath))block{


AVURLAsset * videoAsset = [[AVURLAsset alloc]initWithURL:[NSURL fileURLWithPath:movFilePath]  options:nil];

AVAssetTrack *sourceAudioTrack = [[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];

AVMutableComposition* composition = [AVMutableComposition composition];


AVMutableCompositionTrack *compositionAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                            preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceAudioTrack
                                atTime:kCMTimeZero error:nil];




AVAssetExportSession * assetExport = [[AVAssetExportSession alloc] initWithAsset:composition
                                                                      presetName:AVAssetExportPresetMediumQuality];


NSString *exportPath =  [movFilePath stringByReplacingOccurrencesOfString:@".MOV" withString:@".mp4"];


NSURL * exportUrl = [NSURL fileURLWithPath:exportPath];


assetExport.outputFileType = AVFileTypeMPEG4;
assetExport.outputURL = exportUrl;
assetExport.shouldOptimizeForNetworkUse = YES;
assetExport.videoComposition = [self getVideoComposition:videoAsset composition:composition];

[assetExport exportAsynchronouslyWithCompletionHandler:
 ^(void ) {
     switch (assetExport.status)
     {
         case AVAssetExportSessionStatusCompleted:
             //                export complete
                    if (block) {
                         block(exportPath);
                }
             break;
         case AVAssetExportSessionStatusFailed:
             block(nil);
             break;
         case AVAssetExportSessionStatusCancelled:
            block(nil);
             break;
     }
 }];
}

//get current orientation

  +(AVMutableVideoComposition *) getVideoComposition:(AVAsset *)asset composition:( AVMutableComposition*)composition{
    BOOL isPortrait_ = [self isVideoPortrait:asset];


    AVMutableCompositionTrack *compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];


    AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
    [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration) ofTrack:videoTrack atTime:kCMTimeZero error:nil];

    AVMutableVideoCompositionLayerInstruction *layerInst = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:compositionVideoTrack];

    CGAffineTransform transform = videoTrack.preferredTransform;
    [layerInst setTransform:transform atTime:kCMTimeZero];


    AVMutableVideoCompositionInstruction *inst = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
    inst.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
    inst.layerInstructions = [NSArray arrayWithObject:layerInst];


    AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
    videoComposition.instructions = [NSArray arrayWithObject:inst];

    CGSize videoSize = videoTrack.naturalSize;
    if(isPortrait_) {
        NSLog(@"video is portrait ");
        videoSize = CGSizeMake(videoSize.height, videoSize.width);
    }
    videoComposition.renderSize = videoSize;
    videoComposition.frameDuration = CMTimeMake(1,30);
    videoComposition.renderScale = 1.0;
    return videoComposition;
   }

//get video

+(BOOL) isVideoPortrait:(AVAsset *)asset{
BOOL isPortrait = FALSE;
NSArray *tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
if([tracks    count] > 0) {
    AVAssetTrack *videoTrack = [tracks objectAtIndex:0];

    CGAffineTransform t = videoTrack.preferredTransform;
    // Portrait
    if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)
    {
        isPortrait = YES;
    }
    // PortraitUpsideDown
    if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {

        isPortrait = YES;
    }
    // LandscapeRight
    if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)
    {
        isPortrait = FALSE;
    }
    // LandscapeLeft
    if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0)
    {
        isPortrait = FALSE;
    }
}
return isPortrait;

}

Vinodh
  • 5,262
  • 4
  • 38
  • 68
Jagie
  • 2,190
  • 3
  • 27
  • 25
  • Hi @Jagie thanks for the sample code. I got this working except when using the front camera the exported video is only shown half. Which means I need to move it with translation down along the y-axis. This works however fine if I use the back camera. Did you have similar issue with the front camera? – doorman Mar 14 '16 at 18:48
  • When i tried your code, it gives me an error "The video could not be composed." – Hong Zhou Aug 27 '16 at 10:13
7

In Apple's documentation here it states:

Clients may now receive physically rotated CVPixelBuffers in their AVCaptureVideoDataOutput -captureOutput:didOutputSampleBuffer:fromConnection: delegate callback. In previous iOS versions, the front-facing camera would always deliver buffers in AVCaptureVideoOrientationLandscapeLeft and the back-facing camera would always deliver buffers in AVCaptureVideoOrientationLandscapeRight. All 4 AVCaptureVideoOrientations are supported, and rotation is hardware accelerated. To request buffer rotation, a client calls -setVideoOrientation: on the AVCaptureVideoDataOutput's video AVCaptureConnection. Note that physically rotating buffers does come with a performance cost, so only request rotation if it's necessary. If, for instance, you want rotated video written to a QuickTime movie file using AVAssetWriter, it is preferable to set the -transform property on the AVAssetWriterInput rather than physically rotate the buffers in AVCaptureVideoDataOutput.

So the posted solution by Aaron Vegh that uses an AVAssetExportSession works, but is not needed. Like the Apple doc's say, if you'd like to have the orientation set correctly so that it plays in non-apple quicktime players like VLC or on the web using Chrome, you must set the video orientation on the AVCaptureConnection for the AVCaptureVideoDataOutput. If you try to set it for the AVAssetWriterInput you will get an incorrect orientation for players like VLC and Chrome.

Here is my code where I set it during setting up the capture session:

// DECLARED AS PROPERTIES ABOVE
@property (strong,nonatomic) AVCaptureDeviceInput *audioIn;
@property (strong,nonatomic) AVCaptureAudioDataOutput *audioOut;
@property (strong,nonatomic) AVCaptureDeviceInput *videoIn;
@property (strong,nonatomic) AVCaptureVideoDataOutput *videoOut;
@property (strong,nonatomic) AVCaptureConnection *audioConnection;
@property (strong,nonatomic) AVCaptureConnection *videoConnection;
------------------------------------------------------------------
------------------------------------------------------------------

-(void)setupCaptureSession{
// Setup Session
self.session = [[AVCaptureSession alloc]init];
[self.session setSessionPreset:AVCaptureSessionPreset640x480];

// Create Audio connection ----------------------------------------
self.audioIn = [[AVCaptureDeviceInput alloc]initWithDevice:[self getAudioDevice] error:nil];
if ([self.session canAddInput:self.audioIn]) {
    [self.session addInput:self.audioIn];
}

self.audioOut = [[AVCaptureAudioDataOutput alloc]init];
dispatch_queue_t audioCaptureQueue = dispatch_queue_create("Audio Capture Queue", DISPATCH_QUEUE_SERIAL);
[self.audioOut setSampleBufferDelegate:self queue:audioCaptureQueue];
if ([self.session canAddOutput:self.audioOut]) {
    [self.session addOutput:self.audioOut];
}
self.audioConnection = [self.audioOut connectionWithMediaType:AVMediaTypeAudio];

// Create Video connection ----------------------------------------
self.videoIn = [[AVCaptureDeviceInput alloc]initWithDevice:[self videoDeviceWithPosition:AVCaptureDevicePositionBack] error:nil];
if ([self.session canAddInput:self.videoIn]) {
    [self.session addInput:self.videoIn];
}

self.videoOut = [[AVCaptureVideoDataOutput alloc]init];
[self.videoOut setAlwaysDiscardsLateVideoFrames:NO];
[self.videoOut setVideoSettings:nil];
dispatch_queue_t videoCaptureQueue =  dispatch_queue_create("Video Capture Queue", DISPATCH_QUEUE_SERIAL);
[self.videoOut setSampleBufferDelegate:self queue:videoCaptureQueue];
if ([self.session canAddOutput:self.videoOut]) {
    [self.session addOutput:self.videoOut];
}

self.videoConnection = [self.videoOut connectionWithMediaType:AVMediaTypeVideo];
// SET THE ORIENTATION HERE -------------------------------------------------
[self.videoConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
// --------------------------------------------------------------------------

// Create Preview Layer -------------------------------------------
AVCaptureVideoPreviewLayer *previewLayer = [[AVCaptureVideoPreviewLayer alloc]initWithSession:self.session];
CGRect bounds = self.videoView.bounds;
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
previewLayer.bounds = bounds;
previewLayer.position=CGPointMake(CGRectGetMidX(bounds), CGRectGetMidY(bounds));
[self.videoView.layer addSublayer:previewLayer];

// Start session
[self.session startRunning];

}

Kazzin
  • 181
  • 3
  • 8
  • I'd love to use this kind of solution. It would address a lot of problems. The docs hint at "performance cost". Did this cost turn out to be significant in your experience? I'm curious about presets with the highest bit rates (e.g. HD resolution at 120 frames per second on an iPhone 5s)? – otto Jan 29 '14 at 15:33
3

In case anyone else is looking for this answer as well, this is the method that I cooked up (modified a bit to simplify):

- (void)encodeVideoOrientation:(NSURL *)anOutputFileURL
{
CGAffineTransform rotationTransform;
CGAffineTransform rotateTranslate;
CGSize renderSize;

switch (self.recordingOrientation)
{
    // set these 3 values based on orientation

}


AVURLAsset * videoAsset = [[AVURLAsset alloc]initWithURL:anOutputFileURL options:nil];

AVAssetTrack *sourceVideoTrack = [[videoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
AVAssetTrack *sourceAudioTrack = [[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];

AVMutableComposition* composition = [AVMutableComposition composition];

AVMutableCompositionTrack *compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceVideoTrack
                                atTime:kCMTimeZero error:nil];
[compositionVideoTrack setPreferredTransform:sourceVideoTrack.preferredTransform];

AVMutableCompositionTrack *compositionAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                            preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceAudioTrack
                                atTime:kCMTimeZero error:nil];



AVMutableVideoCompositionInstruction *instruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
AVMutableVideoCompositionLayerInstruction *layerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:compositionVideoTrack];
[layerInstruction setTransform:rotateTranslate atTime:kCMTimeZero];

AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
videoComposition.frameDuration = CMTimeMake(1,30);
videoComposition.renderScale = 1.0;
videoComposition.renderSize = renderSize;
instruction.layerInstructions = [NSArray arrayWithObject: layerInstruction];
instruction.timeRange = CMTimeRangeMake(kCMTimeZero, videoAsset.duration);
videoComposition.instructions = [NSArray arrayWithObject: instruction];

AVAssetExportSession * assetExport = [[AVAssetExportSession alloc] initWithAsset:composition
                                                                      presetName:AVAssetExportPresetMediumQuality];

NSString* videoName = @"export.mov";
NSString *exportPath = [NSTemporaryDirectory() stringByAppendingPathComponent:videoName];

NSURL * exportUrl = [NSURL fileURLWithPath:exportPath];

if ([[NSFileManager defaultManager] fileExistsAtPath:exportPath])
{
    [[NSFileManager defaultManager] removeItemAtPath:exportPath error:nil];
}

assetExport.outputFileType = AVFileTypeMPEG4;
assetExport.outputURL = exportUrl;
assetExport.shouldOptimizeForNetworkUse = YES;
assetExport.videoComposition = videoComposition;

[assetExport exportAsynchronouslyWithCompletionHandler:
 ^(void ) {
     switch (assetExport.status)
     {
         case AVAssetExportSessionStatusCompleted:
             //                export complete
             NSLog(@"Export Complete");
             break;
         case AVAssetExportSessionStatusFailed:
             NSLog(@"Export Failed");
             NSLog(@"ExportSessionError: %@", [assetExport.error localizedDescription]);
             //                export error (see exportSession.error)
             break;
         case AVAssetExportSessionStatusCancelled:
             NSLog(@"Export Failed");
             NSLog(@"ExportSessionError: %@", [assetExport.error localizedDescription]);
             //                export cancelled
             break;
     }
 }];

}

This stuff is poorly documented, unfortunately, but by stringing examples together from other SO questions and reading the header files, I was able to get this working. Hope this helps anyone else!

Aaron Vegh
  • 5,217
  • 7
  • 48
  • 75
2

Use these below method to set correct orientation according to video asset orientation in AVMutableVideoComposition

-(AVMutableVideoComposition *) getVideoComposition:(AVAsset *)asset
{
  AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
  AVMutableComposition *composition = [AVMutableComposition composition];
  AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
  CGSize videoSize = videoTrack.naturalSize;
  BOOL isPortrait_ = [self isVideoPortrait:asset];
  if(isPortrait_) {
      NSLog(@"video is portrait ");
      videoSize = CGSizeMake(videoSize.height, videoSize.width);
  }
  composition.naturalSize     = videoSize;
  videoComposition.renderSize = videoSize;
  // videoComposition.renderSize = videoTrack.naturalSize; //
  videoComposition.frameDuration = CMTimeMakeWithSeconds( 1 / videoTrack.nominalFrameRate, 600);

  AVMutableCompositionTrack *compositionVideoTrack;
  compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
  [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration) ofTrack:videoTrack atTime:kCMTimeZero error:nil];
  AVMutableVideoCompositionLayerInstruction *layerInst;
  layerInst = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
  [layerInst setTransform:videoTrack.preferredTransform atTime:kCMTimeZero];
  AVMutableVideoCompositionInstruction *inst = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
  inst.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
  inst.layerInstructions = [NSArray arrayWithObject:layerInst];
  videoComposition.instructions = [NSArray arrayWithObject:inst];
  return videoComposition;
}


-(BOOL) isVideoPortrait:(AVAsset *)asset
{
  BOOL isPortrait = FALSE;
  NSArray *tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
  if([tracks    count] > 0) {
    AVAssetTrack *videoTrack = [tracks objectAtIndex:0];

    CGAffineTransform t = videoTrack.preferredTransform;
    // Portrait
    if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)
    {
        isPortrait = YES;
    }
    // PortraitUpsideDown
    if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {

        isPortrait = YES;
    }
    // LandscapeRight
    if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)
    {
        isPortrait = FALSE;
    }
    // LandscapeLeft
    if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0)
    {
        isPortrait = FALSE;
    }
   }
  return isPortrait;
}
Paresh Navadiya
  • 38,095
  • 11
  • 81
  • 132
1

Since iOS 5 you can request rotated CVPixelBuffers using AVCaptureVideoDataOutput documented here. This gives you the correct orientation without having to reprocess the video again with AVAssetExportSession.

CEarwood
  • 1,685
  • 10
  • 12
0

Here is the latest swift version of @Jagie code.

extension AVURLAsset
{
    func exportVideo(presetName: String = AVAssetExportPresetHighestQuality, outputFileType: AVFileType = .mp4, fileExtension: String = "mp4", then completion: @escaping (URL?) -> Void)
    {
        let filename = url.deletingPathExtension().appendingPathExtension(fileExtension).lastPathComponent
        let outputURL = FileManager.default.temporaryDirectory.appendingPathComponent(filename)
        
        do { // delete old video, if already exists
                try FileManager.default.removeItem(at: outputURL)
        } catch {
            print(error.localizedDescription)
        }
        
        guard let sourceAudioTrack = self.tracks(withMediaType: .audio).first else { return }
        let composition = AVMutableComposition()
        let compositionAudioTrack = composition.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)
        do{
            try compositionAudioTrack?.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: self.duration), of: sourceAudioTrack, at: CMTime.zero)
        }catch(let e){
            print("error: \(e)")
        }
        
        
        if let session = AVAssetExportSession(asset: composition, presetName: presetName) {
            session.outputURL = outputURL
            session.outputFileType = outputFileType
            let start = CMTimeMakeWithSeconds(0.0, preferredTimescale: 0)
            let range = CMTimeRangeMake(start: start, duration: duration)
            session.timeRange = range
            session.shouldOptimizeForNetworkUse = true
            
            session.videoComposition =  getVideoComposition(asset: self, composition: composition)
            
            session.exportAsynchronously {
                switch session.status {
                case .completed:
                    completion(outputURL)
                case .cancelled:
                    debugPrint("Video export cancelled.")
                    completion(nil)
                case .failed:
                    let errorMessage = session.error?.localizedDescription ?? "n/a"
                    debugPrint("Video export failed with error: \(errorMessage)")
                    completion(nil)
                default:
                    break
                }
            }
        } else {
            completion(nil)
        }
    }
    
    
    private func getVideoComposition(asset: AVAsset, composition: AVMutableComposition) -> AVMutableVideoComposition{
        let isPortrait = isVideoPortrait()

        let compositionVideoTrack: AVMutableCompositionTrack = composition.addMutableTrack(withMediaType: .video, preferredTrackID:kCMPersistentTrackID_Invalid)!
        
        let videoTrack:AVAssetTrack = asset.tracks(withMediaType: .video).first!
        do{
        try compositionVideoTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: asset.duration), of: videoTrack, at: CMTime.zero)
        }catch(let e){
            print("Error: \(e)")
        }
        
        let layerInst:AVMutableVideoCompositionLayerInstruction = AVMutableVideoCompositionLayerInstruction(assetTrack: compositionVideoTrack)
        
        let transform = videoTrack.preferredTransform
        layerInst.setTransform(transform, at: CMTime.zero)

        let inst:AVMutableVideoCompositionInstruction = AVMutableVideoCompositionInstruction()
        inst.timeRange = CMTimeRangeMake(start: CMTime.zero, duration: asset.duration);
        inst.layerInstructions = [layerInst]
        
        let videoComposition: AVMutableVideoComposition = AVMutableVideoComposition()
        videoComposition.instructions = [inst]
        
        var videoSize:CGSize = videoTrack.naturalSize;
        if(isPortrait) {
            print("video is portrait")
            videoSize = CGSize(width: videoSize.height, height: videoSize.width)
        }else{
            print("video is landscape")
        }
        
        videoComposition.renderSize = videoSize;
        videoComposition.frameDuration = CMTimeMake(value: 1,timescale: 30);
        videoComposition.renderScale = 1.0;
        return videoComposition;
    }
    
    func isVideoPortrait() -> Bool{
        var isPortrait = false
        let tracks = self.tracks(withMediaType: .video)
        if(tracks.count > 0) {
            let videoTrack:AVAssetTrack = tracks.first!;

            let t:CGAffineTransform = videoTrack.preferredTransform;
            // Portrait
            if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)
            {
                isPortrait = true;
            }
            // PortraitUpsideDown
            if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {

                isPortrait = true;
            }
            // LandscapeRight
            if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)
            {
                isPortrait = false;
            }
            // LandscapeLeft
            if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0)
            {
                isPortrait = false;
            }
        }
        return isPortrait;
    }
}
virtplay
  • 550
  • 7
  • 18