17

I've managed to load a video-track of a movie frame by frame into an OpenGL texture with AVFoundation. I followed the steps described in the answer here: iOS4: how do I use video file as an OpenGL texture? and took some code from the GLVideoFrame sample from WWDC2010 which can be downloaded here.

How do I play the audio-track of the movie synchronously to the video? I think it would not be a good idea to play it in a separate player, but to use the audio-track of the same AVAsset.

AVAssetTrack* audioTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];

I retrieve a videoframe and it's timestamp in the CADisplayLink-callback via

CMSampleBufferRef sampleBuffer = [self.readerOutput copyNextSampleBuffer];
CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );

where readerOutput is of type AVAssetReaderTrackOutput*

How to get the corresponding audio-samples? And how to play them?

Edit:

I've looked around a bit and I think, best would be to use AudioQueue from the AudioToolbox.framework using the approach described here: AVAssetReader and Audio Queue streaming problem

There is also an audio-player in the AVFoundation: AVAudioPlayer. But I don't know exactly how I should pass data to its initWithData-initializer which expects NSData. Furthermore, I don't think it's the best choice for my case because a new AVAudioPlayer-instance would have to be created for every new chunk of audio samples, as I understand it.

Any other suggestions? What's the best way to play the raw audio samples which I get from the AVAssetReaderTrackOutput?

Amir Shabani
  • 3,857
  • 6
  • 30
  • 67
j00hi
  • 5,420
  • 3
  • 45
  • 82
  • Did you ever get anywhere with this? – RyanSullivan May 02 '12 at 16:12
  • My solution was to open the movie with AVAssetReader to grab the frames for OpenGL, and at the same time open the same movie-file again with an AVAudioPlayer and let both of them start simultaneously. It's not a nice solution but audio and video are quite sync. Since then I didn't have any time to work on a better solution. – j00hi May 17 '12 at 09:01
  • I'm also using AVAudioPlayer but the audio sync isn't very good. Having a real solution would be better. – Nicolas Goy Jun 08 '12 at 14:41
  • In my implementation, video and audio are quite good sync. I'm calling prepare on the AVAudioPlayer, and then call play in the AVAudioPlayer at the same time as starting to read samples from the video-track. But actually you're right, this isn't sync by design, although it works quite good for me. – j00hi Jun 08 '12 at 18:40
  • Try shooting in mixed light conditions with the iPhone 5 and you will immediately see some proper sync issues between video and audio using this method. – BlueVoodoo Oct 24 '12 at 15:18
  • Why does it go out of sync when shooting in mixed light conditions? – j00hi Oct 24 '12 at 19:59

1 Answers1

2

You want do do an AV composition. You can merge multiple media sources, synchronized temporally, into one output.

http://developer.apple.com/library/ios/#DOCUMENTATION/AVFoundation/Reference/AVComposition_Class/Reference/Reference.html

Jody Hagins
  • 27,943
  • 6
  • 58
  • 87