6

I'm using AVAssetExportSession to export videos in an iOS app. To render the videos in their correct orientation, I'm using AVAssetTrack's preferredTransform. For some source videos, this property seems to have a wrong value, and the video appears offset or completely black in the result. How can I fix this?

Theo
  • 3,826
  • 30
  • 59

2 Answers2

10

The preferredTransform is a CGAffineTransform. The properties a, b, c, d are concatenations of reflection and rotation matrices, and the properties tx and ty describe a translation. In all cases that I observed with an incorrect preferredTransform, the reflection/rotation part appeared to be correct, and only the translation part contained wrong values. A reliable fix seems to be to inspect a, b, c, d (eight cases in total, each corresponding to a case in UIImageOrientation) and update tx and ty accordingly:

extension AVAssetTrack {
  var fixedPreferredTransform: CGAffineTransform {
    var t = preferredTransform
    switch(t.a, t.b, t.c, t.d) {
    case (1, 0, 0, 1):
      t.tx = 0
      t.ty = 0
    case (1, 0, 0, -1):
      t.tx = 0
      t.ty = naturalSize.height
    case (-1, 0, 0, 1):
      t.tx = naturalSize.width
      t.ty = 0
    case (-1, 0, 0, -1):
      t.tx = naturalSize.width
      t.ty = naturalSize.height
    case (0, -1, 1, 0):
      t.tx = 0
      t.ty = naturalSize.width
    case (0, 1, -1, 0):
      t.tx = naturalSize.height
      t.ty = 0
    case (0, 1, 1, 0):
      t.tx = 0
      t.ty = 0
    case (0, -1, -1, 0):
      t.tx = naturalSize.height
      t.ty = naturalSize.width
    default:
      break
    }
    return t
  }
}
Theo
  • 3,826
  • 30
  • 59
  • I'm seeing a similar problem but it only seemed to appear recently with iOS 14. Before that, `preferredTransform` seemed to work correctly for years. Have we figured out how the incorrect translation is being created for the asset track? I'd like to file a bug with Apple but I don't have a reproducible test case. – otto Dec 10 '20 at 12:48
  • 3
    I managed to reproduce it on iOS 14 as follows: Record a video in portrait orientation using the camera app. Then, crop the video frame to a smaller rect using the Photos app. The resulting video shows the incorrect preferredTransform. I don't understand the technical details, so I don't know if this is actually a bug in the Photos app or in `AVFoundation` (Is the `preferredTransform` part of the video file or is it created by `AVFoundation`, based on other video attributes?). I filed bug FB8762584 which has been labeled with "Potential fix identified". – Theo Dec 12 '20 at 16:06
  • Great test case @Theo. Thank you. We were able to reproduce the issue with those steps. In terms of `preferredTransform` being retrieved vs calculated, that's a good question. I'm not sure. For what it's worth, when our iOS code encounters video made with Android devices, we found that some of the needed translations in the `preferredTransform` could be zero and needed to be fixed in a similar way. – otto Dec 14 '20 at 12:56
0

I ended up doing something slightly more robust I think, I nullified the transform based on where it would end up:

auto naturalFrame = CGRectMake(0, 0, naturalSize.width, naturalSize.height);
auto preferredFrame = CGRectApplyAffineTransform(naturalFrame, preferredTransform);
preferredTransform.tx -= preferredFrame.origin.x;
preferredTransform.ty -= preferredFrame.origin.y;

Note that you can't just apply the transform on (0, 0) since CGRect.origin takes into account things like flipping.

Barak Yoresh
  • 279
  • 2
  • 4