24

I'm trying to display the contents of a video file (let's just say without the audio for now) onto a UV mapped 3D object in OpenGL. I've done a fair bit in OpenGL but have no idea where to begin in video file handling, and most of the examples out there seems to be for getting video frames from cameras, which is not what I'm after.

At the moment I feel if I can get individual frames of the video as CGImageRef I'd be set, so I'm wondering how to do this? Perhaps there are even be better ways to do this? Where should I start and what's the most straight forward file format for video playback on iOS? .mov?

Billy
  • 1,374
  • 1
  • 17
  • 25

2 Answers2

33

Apologies; typing on an iPhone so I'll be a little brief.

Create an AVURLAsset with the URL of your video - which can be a local file URL if you like. Anything QuickTime can do is fine, so MOV or M4V in H.264 is probably the best source.

Query the asset for tracks of type AVMediaTypeVideo. You should get just one unless your source video has multiple camera angles of something like that, so just taking objectAtIndex:0 should give you the AVAssetTrack you want.

Use that to create an AVAssetReaderTrackOutput. Probably you want to specify kCVPixelFormatType_32BGRA.

Create an AVAssetReader using the asset; attach the asset reader track output as an output. And call startReading.

Henceforth you can call copyNextSampleBuffer on the track output to get new CMSampleBuffers, putting you in the same position as if you were taking input from the camera. So you can lock that to get at pixel contents and push those to OpenGL via Apple's BGRA extension.

Tommy
  • 99,986
  • 12
  • 185
  • 204
  • Also thanks. This all works perfectly and you can get out very high FPS (depending on the video data rate). – Mike Weller Jun 30 '11 at 05:26
  • Another thanks. This worked well for me too. Just note that you have to handle video playback frame rate syncing yourself. – Chris Miles Aug 10 '11 at 02:56
  • I created and example of loading a OpenGL ES 2.0 texture with an alpha channel based on http://www.raywenderlich.com/4404/opengl-es-2-0-for-iphone-tutorial-part-2-textures but with an animated fish instead of a static image here http://iosgraphicsanimation.wordpress.com/2013/06/15/load-opengl-textures-with-alpha-channel-on-ios/. Using AVAsset is fine for 24BPP video, but there is no way to use 32BPP data with an alpha channel using that approach. – MoDJ Jun 21 '13 at 03:13
  • Would this work with movies with alpha channels? Are there any codecs supported by the mov format that would allow for this? – jowie Sep 26 '13 at 14:53
  • Jowie, How about you click on the link in my comment above which explains exactly how to load an alpha channel video? This of course does not work with h.264 since it does not support an alpha channel? – MoDJ Dec 15 '14 at 06:07
  • did someone successfully succeed do play the ios video under openGL? if yes would he mind to share the code ? me i didn't yet succeed :( thanks you by advance ! – zeus Feb 06 '17 at 22:31
-1

You're probably going to have to use a player layer and flatten its contents into a bitmap context. See the documentation for AVPlayerLayer. The performance might be very poor though.

Steven Kramer
  • 8,473
  • 2
  • 37
  • 43