I have a working answer for this, relying on AppleScript to query the QuickTime Player. This requires a few hoops for a hardened-runtime Sandboxed app, including entitlements as well as a special plist key that prompts the user to grant access to your app to script QuickTime Player.
If none of that is a deal-breaker, then here's a small class that detects detects whether QuickTime Player is running and, if so, sends an AppleScript to query for the "Screen Recording" document. If screen recording is in progress (not just open pending recording), the document's current time will be greater than 0 and the script result will be true, otherwise it will be false. There's also a check to see if QuickTime Player is running before the script is attempted; if not it just returns false.
import Cocoa
class ScreenRecordingCheck {
static let qt_query_applescript = """
tell application id "com.apple.QuickTimePlayerX"
set isScreenRecording to false
repeat with doc in documents
set docname to name of doc
if docname is "Screen Recording" then
set doctime to current time of doc
set isScreenRecording to doctime is greater than 0.0
end if
end repeat
set result to isScreenRecording
end tell
"""
var scriptObj: NSAppleScript?
init() {
self.scriptObj = NSAppleScript(source: ScreenRecordingCheck.qt_query_applescript)
}
func isScreenRecordingActive() -> Bool {
var result = false
if !self.isQuicktimePlayerRunning() {
return false
}
if let script = self.scriptObj {
var errDictionary: NSDictionary?
let scriptResult = script.executeAndReturnError(&errDictionary)
if let err = errDictionary {
Swift.print("Error: " + String(describing:err))
} else {
result = scriptResult.booleanValue
}
}
return result
}
func isQuicktimePlayerRunning() -> Bool {
let qtapps = NSRunningApplication.runningApplications(withBundleIdentifier: "com.apple.QuickTimePlayerX")
return !qtapps.isEmpty
}
}
To enable this in a Sandboxed app you'll need this key added to your app's Info.plist:
<key>NSAppleEventsUsageDescription</key>
<string>Please give ScreenRecordingDetector access to Quicktime Player via Apple Script.</string>
...and this entry in your app's .entitlements file:
<key>com.apple.security.temporary-exception.apple-events</key>
<string>com.apple.QuickTimePlayerX</string>
in addition to checking the Apple Events checkbox under the app target's Signing & Capabilities / Hardened Runtime.