AVPlayer playback fails while AVAssetExportSession is active as of iOS 10

Question

AVPlayer will randomly just play audio, not showing the video track...

Playing video correctly with AVPlayer while having an active compression session of AVAssetExportSession:exportAsynchronouslyWithCompletionHandler is now failing. It seems it causes some instability that persists until the app goes into the background and comes back.

Examining AVURLAsset when video playback fails, there is a video track even though there is no video output. I also don't get any errors from the compressor or the other playback components... The problem happens even for a while after the compression session has returned.

I've tried separate threading in different ways with no success.

It seems that as of iOS 10 there is something going on under the hood that causes some sort of conflict between both processes...

@RhythmicFistman I've not made an example project yet. At the moment I've opted to to cut out the post compression opting for a real time solution to work around the bug. FYI, [This other question](http://stackoverflow.com/questions/39300675/white-video-when-opening-avmutablecomposition-in-instagram?noredirect=1#comment66438880_39300675) seems to be the same bug. — Andres Canella, Sep 19 '16 at 15:34
`[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker error:&error];` Capture works perfectly and the bug is not seen if just capturing, it's the post processing where the problem is when `AVAssetExportSession:exportAsynchronouslyWithCompletionHandler` is called. And the issue is only isolated to iOS 10. — Andres Canella, Sep 19 '16 at 22:50
Out of curiosity can you try adding `AVAudioSessionCategoryOptionMixWithOthers` in with your options? — Rhythmic Fistman, Sep 19 '16 at 22:54
@RhythmicFistman Tried, and no good... I ended up getting rid of AVAssetExportSession all together. I'm using AVCaptureSession with AVAssetWriter to do the image adjustments inline while capturing... iOS 10 is doing something odd under the hood... — Andres Canella, Sep 19 '16 at 23:12
I've contacted Apple DTS, let's see what they say about it. If I get the time to make an example project and replicate the bug I'll log it. — Andres Canella, Sep 19 '16 at 23:26
Just heard from Apple DTS. They confirmed that this probably is an Apple iOS bug. — Andres Canella, Sep 21 '16 at 00:18
I hit this exact problem http://stackoverflow.com/questions/39760147/ios-10-avplayerlayer-doesnt-show-video-after-using-avvideocompositioncoreanima/39780044#39780044 and it seems specific to AVVideoCompositionCoreAnimationTool. I can successfully use AVAssetExportSession with a custom video compositor, but using the CA tool makes everything go wrong. — Sami Samhuri, Sep 29 '16 at 21:33
@SamiSamhuri All part of the same internal AV issue.... I also worked around it by removing 'AVAssetExportSession' all together. I put in a bit more work into the camera capture process and did final formatting in real time using 'AVAssetWriter". — Andres Canella, Sep 29 '16 at 22:16
@AndresCanella I worked up a small sample project that allows you to easily reproduce the buggy behavior. Let me know if it'd help you in your talks with Apple. — Clay Garrett, Oct 28 '16 at 16:10
@kleezy Great! Can you put it up on github and share it here? I'm not the owner of the bug report on Apple. You can search for that ticket and post it there too. My problem was mission critical so I could not wait for a fix as I mentioned on my Solution below. I'm less active on this now as I completely removed all use of `AVAssetExportSession`. — Andres Canella, Oct 28 '16 at 16:29
You bet. Here's a link to a sample project that demonstrates the bug: https://github.com/claygarrett/avplayer-bug — Clay Garrett, Oct 28 '16 at 17:44
@SamiSamhuri how did you do to use AVVideoCompositionCoreAnimationTool and avoid this problem? need your help :) — Sam, Nov 09 '16 at 09:10

Andres Canella · Answer 1 · 2016-09-29T22:19:17.720

8

Just heard from Apple DTS. They also agree this points to an Apple iOS bug and asked me to log it.

I cut out usage of AVAssetExportSession altogether and it solved the issue. So calling AVAssetExportSession in combination with other AV methods is what causes the iOS instability.

In my case I was usingAVAssetExportSession for post capture compression. So instead I used AVAssetWriter to sample each frame in real-time to the format I needed...

This should be fixed soon by Apple, hopefully.

edited Sep 29 '16 at 22:19

answered Sep 21 '16 at 00:33

Andres Canella

3,706
1
35
47

If someone finds a workaround or hears back from Apple about this, can they leave a comment how they did it? I had to set the video background as a message telling the user to close and open the app, which is a really horrible experience. – Ethan Sep 29 '16 at 00:27
How are you using 'AVPlayer' and 'AVAssetExportSession'? – Andres Canella Sep 29 '16 at 18:12
I'm using AVExportSession to join a bunch of videos together, and then playing the resulting video using AVPlayer – Ethan Sep 30 '16 at 05:42
I can confirm that that is what I was using that caused the problem. I found that if you add a short space after AVExportSession has finished processing before you start playing that might fix the issue. – Andres Canella Oct 02 '16 at 00:09
@AndresCanella, what short space are you talking about? I added a 2 seconds delay before playing, but I am still having this issue (it's true, not that often, but still annoying). Sometimes it needs at least 10 seconds to play correctly. Did you find any other workaround? Can you please share the Apple response link? Thanks :) – Dan Bodnar Oct 09 '16 at 18:35
Hey @DanBodnar. The "short space" was a few seconds. My workaround was to not use AVAssetExportSession all together as I mentioned on this answer. I don't have a link, this was email which just pointed at it being likely a iOS bug. – Andres Canella Oct 10 '16 at 17:32
@AndresCanella any idea when the bug will be fixed? – Sam Oct 12 '16 at 07:53
@Sam Apple said nothing beyond their suspicion that this is an iOS AV bug. – Andres Canella Oct 13 '16 at 17:31
@AndresCanella thanks. Same for me I opened a bug report. They said it was a duplicate and will be closed. I said this answer don't help at all. They replied saying they are working on the issue but no information on when it's gonna be fixed. – Sam Oct 18 '16 at 16:22
@Sam lol yes I figured it would take a long time to get fixed, and that it is deep where we can't get to... thanks for sharing. Looks like this is still the only option currently. – Andres Canella Oct 19 '16 at 00:56
@AndresCanella lol no updates from apple and the problem persists. Please Andres could you post a solution? – Sam Nov 09 '16 at 09:08
@AndresCanella how would you use AVAssetWriter instead of AVAssetExportSession ? Really need help on that one :) – Sam Nov 09 '16 at 09:09

score 2 · Answer 2 · answered Sep 30 '16 at 08:20

I found a solution to my problem. Like Sami said, the issue appears to be in AVVideoCompositionCoreAnimationTool which I was using to watermark my video. I shifted to using a CIFilter, which actually had cleaner code anyway.

I removed everything with CoreAnimationTool and used this (mixComposition is my AVMutableComposition):

let watermarkFilter = CIFilter(name: "CISourceOverCompositing")!
let watermarkImage = CIImage(image: #imageLiteral(resourceName: "watermark"))!
let videoComposition = AVVideoComposition(asset: mixComposition) { (filteringRequest) in
    let source = filteringRequest.sourceImage.clampingToExtent()
    watermarkFilter.setValue(source, forKey: "inputBackgroundImage")
    let transform = CGAffineTransform(translationX: filteringRequest.sourceImage.extent.width - watermarkImage.extent.width - 2, y: 0)
    watermarkFilter.setValue(watermarkImage.applying(transform), forKey: "inputImage")
    filteringRequest.finish(with: watermarkFilter.outputImage!, context: nil)
}

and then in the AVAssetExportSession added this:

exporter!.videoComposition = videoComposition

Hope that helps somebody!

Are your still using AVAssetExportSession:exportAsynchronouslyWithCompletionHandl‌er ? — Dan Bodnar, Oct 10 '16 at 09:00
Any chance you managed to solve the rotation issue with a custom AVVideoCompositing clas? It's not respecting the asset transform, all the CIFilters in the request are always in landscape... We stuck on this for over a week ah — Roi Mulia, Dec 12 '18 at 12:03

AVPlayer playback fails while AVAssetExportSession is active as of iOS 10

2 Answers2

Linked