23

I want to write my own camera filters for videochat, and ideally apply them in any/all of the popular videochat applications (Zoom, Hangouts, Skype, etc.). The way I imagine this working is to write a macOS application that reads the camera feed, applies my filters, and exposes an additional virtual camera. This virtual camera could then be selected in whichever videochat application.

I've spent many hours researching how to do this and I'm still not clear if it's even possible with modern macOS APIs. There are a few similar questions on StackOverflow (e.g. here, here), but they are either unanswered or very old. I'm hoping this question will collect advice/links/ideas in the right direction for how to do this as of 2020.

Here's what I got so far:

  • There's a popular tool in the live streaming community called OBS Studio. It captures input from different sources (camera, desktop, etc.), has a plugin system for applying effects, and then streams the output to popular services (e.g. Twitch). However, there is no functionality to expose the stream as a virtual camera on macOS. In discussions about this (thread, thread), folks talk about a tool called Syphon and a tool called CamTwist.

  • Unfortunately, Syphon doesn't expose a virtual camera anymore: "SyphonInject NO LONGER WORKS IN macOS 10.14 (Mojave). Apple closed up the loophole that allows scripting additions in global directories to load into any process. Trying to inject into any process will silently fail. It will work if SIP is disabled, but that's a terrible idea and I'm not going to suggest or help anyone do that."

  • Fortunately, CamTwist works. I got it running on my macOS Catalina, applied some of its builtin effects on my camera stream, and saw it show up as a new camera in my Hangouts settings (after restarting Chrome). This was encouraging.

  • Unfortunately, CamTwist is rather old and not well maintained. It uses Quartz Composer for implementing effects, but Quartz Composer was deprecated by Apple and it's probably living its last days in Catalina.

  • The macOS SDK used to have an API called CoreMediaIO, which might have been the way to expose a virtual camera, but this API was also deprecated. It's not clear if/what is a modern alternative.

  • I guess another way of asking this whole question is: how is CamTwist implemented, how come it still works in macOS Catalina, and how would you implement the same thing in 2020?

Anything that sheds some light on all of this would be highly appreciated!

user18184
  • 488
  • 5
  • 14
  • fwiw i've been using OBS + CamTwist for the past week or so since i've gotten a mac. It's worked fine with zoom/etc. until just today actually (a few hours ago) when i installed a new osx update and it broke :| edit: actually, it appears to have only broken on zoom. must be the new zoom update today. all others are fine :) – Justin L. Apr 09 '20 at 06:12
  • edit: looks like it works for google meets and back into OBS, but it doesn't show up in photobooth. might be an osx api issue – Justin L. Apr 09 '20 at 06:18
  • 1
    Check this out . https://github.com/satoshi0212/VirtualCameraSample – Syed Sadrul Ullah Sahad Dec 30 '20 at 06:13
  • Very late reply, but I just want this directory to be on this page for future reference. There is a directory on MacOS at `/Library/CoreMediaIO/Plug-Ins/DAL/` that contains `.plugin` files for virtual cameras. I.e. `SnapCamera.plugin`. – twe4ked Feb 13 '23 at 23:51

3 Answers3

2

I also want to create own camera filter like Snap Camera. So I researched around CoreMediaIO and Syphon. Did you check this Github project? https://github.com/lvsti/CoreMediaIO-DAL-Example

This repository started off as a fork of the official CoreMediaIO sample code by Apple. You know, the original code didn't age well since it was last updated in 2012. So the owner of the repository changed to make it compile on modern systems.

And you can know that the code works in macOS 10.14 (Mojave) to see the following issue. https://github.com/lvsti/CoreMediaIO-DAL-Example/issues/4

Actually I have not created the camera filter yet because I don't know how to send images to virtual camera that builded by CoreMediaIO. I would like to know more information. If you know please tell me.

tommy19970714
  • 87
  • 1
  • 7
  • And you can check the following the github project. https://github.com/johnboiles/coremediaio-dal-minimal-example Now it does not work. But I believe that it will work soon. – tommy19970714 Apr 12 '20 at 18:43
1

There is a new API in CoreMediaIO announced at WWDC 2022:

Discover how you can use Core Media IO to easily create macOS system extensions for software cameras, hardware cameras, and creative cameras. We'll introduce you to our modern replacement for legacy DAL plug-ins — these extensions are secure, fast, and fully-compatible with any app that uses a camera input. We'll take you through the Core Media IO APIs and share how they can support camera manufacturers, video conferencing apps with special effects features, creative app ideas, and more.

https://developer.apple.com/videos/play/wwdc2022/10022/

nSquid
  • 763
  • 1
  • 8
  • 15
0

CamTwist uses CoreMedioIO. What makes you think that's deprecated? Looking at the headers in the 10.15 SDK, I see no indication that it's deprecated. There were updates as recently as 10.14.

Ken Thomases
  • 88,520
  • 7
  • 116
  • 154
  • This [developer page](https://developer.apple.com/library/archive/samplecode/CoreMediaIO/Introduction/Intro.html) says "This document is no longer being updated" and I couldn't find an equivalent in the latest documentation. Also, a Google search for "CoreMediaIO" shows discussions (from SO and other forums) about the example code being broken and other confusion. If CoreMediaIO is the way to go, is there a good place to learn how to use it? Thanks! – user18184 Apr 09 '20 at 04:10
  • 1
    The archival of documentation is confusing. Apple changed the style of their documentation and archived all documentation using the old style, regardless of whether its content is still relevant or not. The headers are definitive. I'm not sure about the example code not working. The [question you linked to ](https://stackoverflow.com/questions/56157333/what-is-coremediaio-dal-virtual-camera-alternative-in-macos-mojave) has been self-answered that the poster figured out the problem and got things working, so there's that. Also, as I say, CamTwist, which you report works, uses CoreMediaIO. – Ken Thomases Apr 09 '20 at 13:49