7

New to Mac OS X, familiar with Windows. Windows has DirectShow, a good number of built-in filters, COM programming, and GraphEdit for very fast prototyping and snooping on the graphs you've constructed in code.

I'm now about to go to the Mac to work with cameras, webcams, microphones, color spaces, files, splitting, synchronization, rendering, file reading, file saving, and many of things I've come to take for granted with DirecShow when putting together applications for live performance. On the Mac side, so far I've found ... nothing! Either I don't know where to look or I'm having the toughest time tying the Mac's reputation for its ease of handling media with a coherent programmatic ability to get in there and start messin' with media manipulatin' building blocks.

I've seen some weak suggestions to use gstreamer or some library for QT but I can't bring myself to believe that this is the Apple way to go. And I've come across some QuickTime documentation but I'm not looking to do transitions, sprites, broadcasting, ...

Having a brain trained on DirectShow means I don't even know how Apple thinks about providing DirectShow-like functionality. That means I don't know the right keywords and don't even know where to look. Books? Bought a few. Now I might be able to write some code that can edit your sister's wedding video (if I can't make decent headway on this topic I may next be asking what that'd be worth to you), but for identifying what filters are available and how to string them together ... nothing. Suggestions?

Joe Raglenid
  • 71
  • 1
  • 2
  • I believe QuickTime is it. QT is what's underlying [the hood in iTunes.](http://developer.apple.com/quicktime/) Check out this sample code for [full-screen video playback.](http://developer.apple.com/library/mac/#samplecode/MyMediaPlayer/Introduction/Intro.html) – Stephen Furlani Apr 08 '11 at 18:31
  • 1
    Nope! I'm wrong. Try [Core Video](http://developer.apple.com/library/mac/#documentation/GraphicsImaging/Conceptual/CoreVideo/CVProg_Intro/CVProg_Intro.html) that mentions filters. – Stephen Furlani Apr 08 '11 at 18:33
  • Thanks for the pointers. My initial look at Core Video may have misguided me. It seemed to say it was good for extracting a frame from video, not the direction I was going in. – Joe Raglenid Apr 09 '11 at 01:15

1 Answers1

10

Video handling is going through a huge transition on the Mac at the moment. QuickTime is very old, but also big and powerful, so it's been undergoing an incremental replacement process for the past 5 years or so.

That said, QTKit is the QuickTime subset (capture, playback, format conversion and basic video editing) which is supported going forward. The legacy QuickTime APIs are still there for the moment, and probably will remain at least until its major features are available elsewhere, but are 32-bit only. For some involved video stuff you may end up needing to use it in places.

At the moment, iOS is ahead of the Mac because it could start from scratch with AV Foundation. The future of the Mac media frameworks will probably either be AV Foundation directly (with QTKit being a lightweight shim over the top) or an extension of QTKit that looks very similar.

For audio there's Core Audio which is on Mac and iOS and isn't going away any time soon. It's quite powerful but somewhat obtuse in places. Luckily online support is very good; the mailing list is an essential resource.

For filters and frame-level processing you've got Core Video as someone else mentioned, as well as Core Image. For motion graphics there's Quartz Composer which includes a graphical editor and a plugin architecture to add your own patches. For programmatic procedural animation and easily mixing rendering models (OpenGL, Quartz, video, etc.) there's Core Animation.

In addition to all of these, of course there's no reason you can't use open source libraries where the built-in stuff doesn't do what you want.

To address your comment below:

In QuickTime (and QTKit), individual data types like audio and video are represented as tracks. It may not be immediately clear that QuickTime can open audio as well as video file formats. A common way to combine audio and video would be:

  1. Create a QTMovie with your video file.
  2. Create a QTMovie with your audio file.
  3. Take the QTTrack object representing the audio and add it to the QTMovie with the video in it.
  4. Flatten the movie, so it doesn't simply contain a reference to the other movie but actually contains the audio data.
  5. Write the movie to disk.

Here's an example from Blender. You'll see how the A/V muxing is done in the end_qt function. There's also some use of Core Audio in there (AudioConverter*). (There's some classic QuickTime export code in quicktime_export.c but it doesn't seem to do audio.)

Nicholas Riley
  • 43,532
  • 6
  • 101
  • 124
  • First, thank you for such comprehensive information. I'm still trying to get my mind around it. Set aside for the moment that one technology is taking the place of the other. There seem to be so many moving pieces, so many distinct areas to have to pay attention to. In this way the philosophy seems very different to DirectShow. Where would I look if I wanted to process some audio, process some video, then join them and make sure they were synchronized based on capture time? That's obvious in DirectShow. With this kind of organization on the Mac side, it superficially seems like an afterthought – Joe Raglenid Apr 09 '11 at 01:14
  • Please see my edited answer... my response was too long for a comment. – Nicholas Riley Apr 09 '11 at 02:00