How to use AVAssetReader and AVAssetWriter for multiple tracks (audio and video) simultaneously?

Question

I know how to use AVAssetReader and AVAssetWriter, and have successfully used them to grab a video track from one movie and transcode it into another. However, I'd like to do this with audio as well. Do I have to create and AVAssetExportSession after I've done with the initial transcode, or is there some way to switch between tracks while in the midst of a writing session? I'd hate to have to deal with the overhead of an AVAssetExportSession.

I ask because, using the pull style method - while ([assetWriterInput isReadyForMoreMediaData]) {...} - assumes one track only. How could it be used for more than one track, i.e. both an audio and a video track?

Would you be able to point me in the right direction to learn how to transcode using AVAssetWriter? Just looking to transcode to lower bitrate and resolution? — Ryan, Jun 01 '12 at 17:51

score 8 · Answer 1 · answered Apr 09 '11 at 01:20

AVAssetWriter will automatically interleave requests on its associated AVAssetWriterInputs in order to integrate different tracks into the output file. Just add an AVAssetWriterInput for each of the tracks that you have, and then call requestMediaDataWhenReadyOnQueue:usingBlock: on each of your AVAssetWriterInputs.

Here's a method I have that calls requestMediaDataWhenReadyOnQueue:usingBlock:. I call this method from a loop over the number of output/input pairs I have. (A separate method is good both for code readability and also because, unlike a loop, each call sets up a separate stack frame for the block.)

You only need one dispatch_queue_t and can reuse it for all of the tracks. Note that you definitely should not call dispatch_async from your block, because requestMediaDataWhenReadyOnQueue:usingBlock: expects the block to, well, block until it has filled in as much data as the AVAssetWriterInput will take. You don't want to return before then.

- (void)requestMediaDataForTrack:(int)i {
  AVAssetReaderOutput *output = [[_reader outputs] objectAtIndex:i];
  AVAssetWriterInput *input = [[_writer inputs] objectAtIndex:i];

  [input requestMediaDataWhenReadyOnQueue:_processingQueue usingBlock:
    ^{
      [self retain];
      while ([input isReadyForMoreMediaData]) {
        CMSampleBufferRef sampleBuffer;
        if ([_reader status] == AVAssetReaderStatusReading &&
            (sampleBuffer = [output copyNextSampleBuffer])) {

          BOOL result = [input appendSampleBuffer:sampleBuffer];
          CFRelease(sampleBuffer);

          if (!result) {
            [_reader cancelReading];
            break;
          }
        } else {
          [input markAsFinished];

          switch ([_reader status]) {
            case AVAssetReaderStatusReading:
              // the reader has more for other tracks, even if this one is done
              break;

            case AVAssetReaderStatusCompleted:
              // your method for when the conversion is done
              // should call finishWriting on the writer
              [self readingCompleted];
              break;

            case AVAssetReaderStatusCancelled:
              [_writer cancelWriting];
              [_delegate converterDidCancel:self];
              break;

            case AVAssetReaderStatusFailed:
              [_writer cancelWriting];
              break;
          }

          break;
        }
      }
    }
  ];
}

Whats with the `[self retain]`? Won't that just keep incrementing the retain count on the object an arbitrary amount? Who is releasing this object? — Robotbugs, Jul 12 '13 at 00:58
The intention may have been for `readingCompleted` to release the object. That being said, it may be unnecessary as the presence of the `[self readingCompleted]` call should cause the runtime to retain `self` automatically. — Fiona Hopkins, Jul 15 '13 at 15:00

score 1 · Answer 2 · answered Mar 14 '11 at 06:36

1

Have you tried using two AVAssetWriterInputs and pushing the samples through a worker queue? Here is a rough sketch.

processing_queue = dispatch_queue_create("com.mydomain.gcdqueue.mediaprocessor", NULL);

[videoAVAssetWriterInput requestMediaDataWhenReadyOnQueue:myInputSerialQueue usingBlock:^{
    dispatch_asyc(processing_queue, ^{process video});
}];

[audioAVAssetWriterInput requestMediaDataWhenReadyOnQueue:myInputSerialQueue usingBlock:^{
    dispatch_asyc(processing_queue, ^{process audio});
}];

answered Mar 14 '11 at 06:36

Steve McFarlin

3,576
1
25
24

Thanks for the suggestion. I've tried it, but couldn't figure out how to properly end the writing session. With one track, I just see if copyNextSampleBuffer returns empty, then end the session. With multiple tracks, I tried creating a flag for when the audio or video was done, and if both were complete I would end the session. However, even though I did a check for the asset reader status being AVAssetReaderStatusReading, I kept getting the error: [AVAssetReaderTrackOutput copyNextSampleBuffer] cannot copy next sample buffer unless the asset reader is in the 'reading' state' – akaru Mar 15 '11 at 08:24
I will bet testing a strategy much like this in the near future. If I find a solution I will post back. – Steve McFarlin Mar 15 '11 at 20:24
Are you calling [AVAssetWriterInput markAsFinished]; in your requestMediaDataWhenReadyOnQueue method? I would not call that method at all, but rather call finishWriting on the AVAssetWriter when everything is complete. – Steve McFarlin Mar 22 '11 at 19:07
sorry for the late reply, I wasn't notified of the comment. I only call finishWriting, just as you're doing. – akaru Apr 07 '11 at 05:26
By the way, your "'reading' state" error may actually be a codec issue. If the inputs and outputs are not compatible formats that's the way it will sometimes surface. – Fiona Hopkins Apr 09 '11 at 01:26

score 0 · Answer 3 · answered Sep 16 '12 at 15:54

You can use dispatch groups!

Check out the AVReaderWriter example for MacOSX...

I am quoting directly from the sample RWDocument.m:

- (BOOL)startReadingAndWritingReturningError:(NSError **)outError
{
    BOOL success = YES;
    NSError *localError = nil;

    // Instruct the asset reader and asset writer to get ready to do work
    success = [assetReader startReading];
    if (!success)
        localError = [assetReader error];
    if (success)
    {
        success = [assetWriter startWriting];
        if (!success)
            localError = [assetWriter error];
    }

    if (success)
    {
        dispatch_group_t dispatchGroup = dispatch_group_create();

        // Start a sample-writing session
        [assetWriter startSessionAtSourceTime:[self timeRange].start];

        // Start reading and writing samples
        if (audioSampleBufferChannel)
        {
            // Only set audio delegate for audio-only assets, else let the video channel drive progress
            id <RWSampleBufferChannelDelegate> delegate = nil;
            if (!videoSampleBufferChannel)
                delegate = self;

            dispatch_group_enter(dispatchGroup);
            [audioSampleBufferChannel startWithDelegate:delegate completionHandler:^{
                dispatch_group_leave(dispatchGroup);
            }];
        }
        if (videoSampleBufferChannel)
        {
            dispatch_group_enter(dispatchGroup);
            [videoSampleBufferChannel startWithDelegate:self completionHandler:^{
                dispatch_group_leave(dispatchGroup);
            }];
        }

        // Set up a callback for when the sample writing is finished
        dispatch_group_notify(dispatchGroup, serializationQueue, ^{
            BOOL finalSuccess = YES;
            NSError *finalError = nil;

            if (cancelled)
            {
                [assetReader cancelReading];
                [assetWriter cancelWriting];
            }
            else
            {
                if ([assetReader status] == AVAssetReaderStatusFailed)
                {
                    finalSuccess = NO;
                    finalError = [assetReader error];
                }

                if (finalSuccess)
                {
                    finalSuccess = [assetWriter finishWriting];
                    if (!finalSuccess)
                        finalError = [assetWriter error];
                }
            }

            [self readingAndWritingDidFinishSuccessfully:finalSuccess withError:finalError];
        });

        dispatch_release(dispatchGroup);
    }

    if (outError)
        *outError = localError;

    return success;
}

Notice the dispatch group notify command at the end to know when all audio/video is finished... this is offscreen rendering... not realtime — Orbitus007, Sep 16 '12 at 15:55
"Offscreen" and "real-time" are not opposites; "rendered" and "off-screen" are (and do). "Real-time" is subjective to an observer, and describes the display rate of video frames as they are acquired from their source. It has no opposite, not even "non-real-time." You wouldn't want a transcoder to work at a real-time pace. Transcoding should happen as quickly as possible; there are dedicated devices for this in the broadcast industry. Nothing goes slower than normal real-time, anyway. Even playback software has to pace video to keep it from displaying faster than it can be perceived. — James Bush, Nov 12 '16 at 14:25

How to use AVAssetReader and AVAssetWriter for multiple tracks (audio and video) simultaneously?

3 Answers3

Linked