Composing Video and Audio - Video's audio is gone

Question

My question is, I am using the function below, to compose a video and audio. I want to keep video's original sound but it goes away somehow, I do not have any clue.

I got this function from this answer

I tried to change volumes right after appending AVMutableCompositionTracks but it did not work

For instance;

mutableVideoCompositionTrack.prefferedVolume = 1.0
mutableAudioCompositionTrack.prefferedVolume = 0.05

But still, all you can hear is only the audio file.

The function;

private func mergeAudioAndVideo(audioUrl: URL, videoUrl: URL, completion: @escaping (Bool)->Void){

    let mixComposition = AVMutableComposition()
    var mutableCompositionVideoTrack : [AVMutableCompositionTrack] = []
    var mutableCompositionAudioTrack : [AVMutableCompositionTrack] = []
    let totalVideoCompositionInstruction = AVMutableVideoCompositionInstruction()

    let videoAsset = AVAsset(url: videoUrl)
    let audioAsset = AVAsset(url: audioUrl)

    mutableCompositionVideoTrack.append(mixComposition.addMutableTrack(withMediaType: AVMediaTypeVideo, preferredTrackID: kCMPersistentTrackID_Invalid))
    mutableCompositionAudioTrack.append(mixComposition.addMutableTrack(withMediaType: AVMediaTypeAudio, preferredTrackID: kCMPersistentTrackID_Invalid))
    mutableCompositionAudioTrack[0].preferredVolume = 0.05
    mutableCompositionVideoTrack[0].preferredVolume = 1.0        


    let videoAssetTrack = videoAsset.tracks(withMediaType: AVMediaTypeVideo)[0]
    let audioAssetTrack = audioAsset.tracks(withMediaType: AVMediaTypeAudio)[0]

    do {
        try mutableCompositionVideoTrack[0].insertTimeRange(CMTimeRangeMake(kCMTimeZero, videoAssetTrack.timeRange.duration), of: videoAssetTrack, at: kCMTimeZero)
        try mutableCompositionAudioTrack[0].insertTimeRange(CMTimeRangeMake(kCMTimeZero, videoAssetTrack.timeRange.duration), of: audioAssetTrack, at: kCMTimeZero)
    }catch{
        print("ERROR#1")
    }

    totalVideoCompositionInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, videoAssetTrack.timeRange.duration)

    let mutableVideoComposition = AVMutableVideoComposition()
    mutableVideoComposition.frameDuration = CMTimeMake(1, 30)
    mutableVideoComposition.renderSize = CGSize(width: 1280, height: 720)

    //exporting

    savePathUrl = try! FileManager.default.url(for: FileManager.SearchPathDirectory.documentDirectory, in: FileManager.SearchPathDomainMask.userDomainMask, appropriateFor: nil, create: true).appendingPathComponent("merged").appendingPathExtension("mov")

    let assetExport = AVAssetExportSession(asset: mixComposition, presetName: AVAssetExportPresetHighestQuality)!
    assetExport.outputFileType = AVFileTypeMPEG4
    assetExport.outputURL = savePathUrl
    assetExport.shouldOptimizeForNetworkUse = true

    do {
        try FileManager.default.removeItem(at: savePathUrl)
    }catch {
        print(error)
    }

    assetExport.exportAsynchronously { 
        switch assetExport.status{
        case .completed:
            print("completed")
            completion(true)
        default:
            print("failed \(assetExport.error!)")
            completion(false)
        }
    }

}

score 4 · Accepted Answer · answered Oct 12 '17 at 10:28

You can adjust volume for video and audio separately @Faruk, Here a is little bit code for that.

        //Extract audio from the video and the music
let audioMix: AVMutableAudioMix = AVMutableAudioMix()
var audioMixParam: [AVMutableAudioMixInputParameters] = []

let assetVideoTrack: AVAssetTrack = assetVideo.tracksWithMediaType(AVMediaTypeAudio)[0]
let assetMusicTrack: AVAssetTrack = assetMusic.tracksWithMediaType(AVMediaTypeAudio)[0]

let videoParam: AVMutableAudioMixInputParameters = AVMutableAudioMixInputParameters(track: assetVideoTrack)
videoParam.trackID = compositionAudioVideo.trackID

let musicParam: AVMutableAudioMixInputParameters = AVMutableAudioMixInputParameters(track: assetMusicTrack)
musicParam.trackID = compositionAudioMusic.trackID

//Set final volume of the audio record and the music
videoParam.setVolume(volumeVideo, atTime: kCMTimeZero)
musicParam.setVolume(volumeAudio, atTime: kCMTimeZero)

//Add setting
audioMixParam.append(musicParam)
audioMixParam.append(videoParam)

//Add audio on final record
//First: the audio of the record and Second: the music
do {
try compositionAudioVideo.insertTimeRange(CMTimeRangeMake(kCMTimeZero, assetVideo.duration), ofTrack: assetVideoTrack, atTime: kCMTimeZero)
} catch _ {
assertionFailure()
}

do {
try compositionAudioMusic.insertTimeRange(CMTimeRangeMake(CMTimeMake(Int64(startAudioTime * 10000), 10000), assetVideo.duration), ofTrack: assetMusicTrack, atTime: kCMTimeZero)
} catch _ {
assertionFailure()
}

//Add parameter
audioMix.inputParameters = audioMixParam

let completeMovie = "\(docsDir)/\(randomString(5)).mp4"
let completeMovieUrl = NSURL(fileURLWithPath: completeMovie)
let exporter: AVAssetExportSession = AVAssetExportSession(asset: composition, presetName: AVAssetExportPresetHighestQuality)!

exporter.outputURL = completeMovieUrl
exporter.outputFileType = AVFileTypeMPEG4
exporter.audioMix = audioMix
exporter.exportAsynchronouslyWithCompletionHandler({ 

switch exporter.status {

case AVAssetExportSessionStatus.Completed:
    print("success with output url \(completeMovieUrl)")
    case  AVAssetExportSessionStatus.Failed:
        print("failed \(String(exporter.error))")
    case AVAssetExportSessionStatus.Cancelled:
        print("cancelled \(String(exporter.error))")
    default:
        print("complete")
    }            
})

}

this is a great answer. Thank you for your time on this. For others, please note that this is not Swift 4, but it easily updates itself. — BennyTheNerd, Mar 13 '19 at 00:44

inokey · Answer 2 · 2019-12-20T09:17:59.413

Here's the combined code of all the answers. It took me a while to decode those answers so I decided to add it here for any future users:

Swift 5:

enum MixError: Error {
   case TimeRangeFailure
   case ExportFailure
}

var selectedVideoLevel = 1.0
var selectedMusicLevel = 1.0

func mix(videoUrl: URL, musicUrl: URL, completion: ((Result<URL, Error>) -> Void)?) {
    let videoAsset = AVAsset(url: videoUrl)
    let musicAsset = AVAsset(url: musicUrl)

    let audioVideoComposition = AVMutableComposition()

    let audioMix = AVMutableAudioMix()
    var mixParameters = [AVMutableAudioMixInputParameters]()

    let videoCompositionTrack = audioVideoComposition
      .addMutableTrack(withMediaType: .video, preferredTrackID: .init())!

    let audioCompositionTrack = audioVideoComposition
      .addMutableTrack(withMediaType: .audio, preferredTrackID: .init())!

    let musicCompositionTrack = audioVideoComposition
      .addMutableTrack(withMediaType: .audio, preferredTrackID: .init())!

    let videoAssetTrack = videoAsset.tracks(withMediaType: .video)[0]
    let audioAssetTrack = videoAsset.tracks(withMediaType: .audio).first
    let musicAssetTrack = musicAsset.tracks(withMediaType: .audio)[0]

    let audioParameters = AVMutableAudioMixInputParameters(track: audioAssetTrack)
    audioParameters.trackID = audioCompositionTrack.trackID

    let musicParameters = AVMutableAudioMixInputParameters(track: musicAssetTrack)
    musicParameters.trackID = musicCompositionTrack.trackID

    audioParameters.setVolume(selectedVideoLevel, at: .zero)
    musicParameters.setVolume(selectedMusicLevel, at: .zero)

    mixParameters.append(audioParameters)
    mixParameters.append(musicParameters)

    audioMix.inputParameters = mixParameters

    /// prevents video from unnecessary rotations
    videoCompositionTrack.preferredTransform = videoAssetTrack.preferredTransform

    do {
      let timeRange = CMTimeRange(start: .zero, duration: videoAsset.duration)

      try videoCompositionTrack.insertTimeRange(timeRange, of: videoAssetTrack, at: .zero)

      if let audioAssetTrack = audioAssetTrack {
        try audioCompositionTrack.insertTimeRange(timeRange, of: audioAssetTrack, at: .zero)
      }

      try musicCompositionTrack.insertTimeRange(timeRange, of: musicAssetTrack, at: .zero)

    } catch {
      completion?(.failure(MixError.TimeRangeFailure)
    }


    let exportUrl = FileManager.default
      .urls(for: .applicationSupportDirectory, in: .userDomainMask).first?
      .appendingPathComponent("\(Date().timeIntervalSince1970)-video.mp4")

    let exportSession = AVAssetExportSession(
      asset: audioVideoComposition,
      presetName: AVAssetExportPresetHighestQuality
    )

    exportSession?.audioMix = audioMix
    exportSession?.outputFileType = .m4v
    exportSession?.outputURL = exportUrl

    exportSession?.exportAsynchronously(completionHandler: {
      guard let status = exportSession?.status else { return }

      switch status {
      case .completed:
        completion?(.success(exportUrl!))
     case .failed:
        completion?(.failure(MixError.ExportError)
      default:
        print(status)

      }

    })

  }

score 3 · Answer 3 · answered Jul 26 '17 at 13:03

3

I figured it out. It seems an AVAsset which loads a video holds the audio and video separately. So you can reach them writing``

videoAsset.tracks(withMediaType: AVMediaTypeAudio)[0] //audio of a video
videoAsset.tracks(withMediaType: AVMediaTypeVideo)[0] //video of a video(without sound)

So I added these lines to the code and it worked!

var mutableCompositionBackTrack : [AVMutableCompositionTrack] = []

mutableCompositionBackTrack.append(mixComposition.addMutableTrack(withMediaType: AVMediaTypeAudio, preferredTrackID: kCMPersistentTrackID_Invalid)) 

try mutableCompositionBackTrack[0].insertTimeRange(CMTimeRangeMake(kCMTimeZero, videoAssetTrack.timeRange.duration), of: backAssetTrack, at: kCMTimeZero)

There is a still missing point that I do not know how to do, and that is setting volumes of these audio assets. I will update this answer as soon as I figure out how.

answered Jul 26 '17 at 13:03

Faruk

2,269
31
42

I want to keep video's audio with my custom sound in one file so it plays both audios while playing final video. is this functionality possible with this? – parth Jul 28 '17 at 14:03
That is exactly what i did here. But as i said, i do no know how to change volume levels of each audio. @parth – Faruk Jul 30 '17 at 06:31
@FarukWell I am also facing the same adjust video's audio volume issue after merging videos into one video. Can you help me achieve this. Thanks! – Anand Gautam Mar 14 '18 at 13:35
Apperantly, the problem was not about the volume but the sound itself actually missing.`assetVideo.tracksWithMediaType(AVMediaTypeAudio)[0]` this piece of code returns an array that contains both video and audio seperately. So you should extract two video files and the sound file that you want to compose. Afterwards, it is all about merging. Please carefully examine the code in the question and the answer of @Vivek Goswami. – Faruk Mar 14 '18 at 13:42
Can check for this: https://stackoverflow.com/questions/51419495/fade-effect-is-not-working-between-videos-while-merging-in-swift3-ios – Anand Gautam Jul 19 '18 at 15:21

Composing Video and Audio - Video's audio is gone

3 Answers3

Linked