Problem summary:
I'm working on my own window manager for macOS and one of the features involves moving the windows of other applications using mouse events and the Accessibility API. I create event taps for key pressed and mouse moved events with CGEvent.tapCreate and after my "move mode" is entered by pressing cmd+w I feed mouse moved deltas (while freezing the cursor sprite itself) to a function that takes the location of the currently focused window, adds the deltas, and sets the position of the window (an AXUIElement) by using AXUIElementSetAttributeValue(windowElement, kAXPositionAttribute as CFString, position). This gets called a lot, once for every mouse moved event, and works fine when I have one display (built in MacBook display). Very snappy, no discernible lag, the window gets moved around as if it were the cursor.
However, when I test with more than one display (I've tested using either or both of a wired Dell monitor and a sidecar iPad), I get very inconsistent behavior and generally a lot of lag during window movement. The movement seems to be mostly correct, but delayed (the window might keep moving for as much as a few seconds after I lift my finger from the trackpad, or keep moving in the opposite direction of where I'm moving my finger for a bit).
Sometimes the movement is snappy and correct on one or more displays (otherwise identical behavior to using a single display), but slow on others. Which displays are affected seems random between launches. On one occasion moving my "Terminal" window was consistently snappy while "Notes" was slow. The most common case is just generally sluggish behavior across all displays. Usually at some point the mouse event tap will stop (I guess taking too long causes this to happen). I could restart it but that doesn't really solve the problem.
What I've tried:
Using Instruments (which I'm a novice at so I may have missed something) & timing code I'm 95% sure the problem is with using the call to AXUIElementSetAttributeValue to set the window position. If I take out all the logic regarding where to set (not getting where the window is currently, etc.) and just set the window to the same location repeatedly in response to mouse events, I can still observe the mouse event tap getting stopped (fails pretty silently). Doing all the logic except the location setting doesn't take long enough to stop the tap.
In researching approaches to accomplish what I want I found the following links:
Move other windows on Mac OS X using Accessibility API
How can I move/resize windows programmatically from another application?
Window move and resize APIs in OS X
- Seem to be Obj-C versions of the same approach I'm using
set the size and position of all windows on the screen in swift
- Swift approach to what I'm doing, very similar to my window movement code
I want to animate the movement of a foreign OS X app's window
- This approach is about custom animating window movement rather than just moving but the asker indicates that they were able to move foreign windows by "getting an
AXUIElementRefthat holds the AXUIElement associated with the focused window, then setting theNSAccessibilityPositionAttribute." I have not been able to find more details about this approach and a lot of NSAccessibility stuff appears to have since been deprecated.
I also looked at various Apple docs including Quartz and of course a ton of Accessibility as well as other less useful sources. Nowhere did I find any mentions of performance discrepancy when using the Accessibility API with multiple displays.
Code:
// CODE FOR MOVING A WINDOW
class WindowTransformer {
var windowElement:AXUIElement?
init?(forWindowWithPid: pid_t) {
// Make Accessibility object for given PID
let accessApp:AXUIElement = AXUIElementCreateApplication(forWindowWithPid)
var windowData:AnyObject?
AXUIElementCopyAttributeValue(accessApp, kAXWindowsAttribute as CFString, &windowData)
windowElement = (windowData as? [AXUIElement])?.first
guard let _ = windowElement else { return nil }
}
func transformWindowWithDeltas(x: CGFloat, y: CGFloat) {
let current = getCurrentWindowPosition()
guard let current = current else { return }
let newX = current.x + x
let newY = current.y + y
setPosition(to: CGPoint(x: newX, y: newY))
}
func getCurrentWindowPosition() -> CGPoint? {
if windowElement == nil { return nil }
var positionData:CFTypeRef?
AXUIElementCopyAttributeValue(windowElement!,
kAXPositionAttribute as CFString,
&positionData)
let currentPos = axValueAsCGPoint(positionData! as! AXValue)
return currentPos
}
func axValueAsCGPoint(_ value: AXValue) -> CGPoint {
var point = CGPoint.zero
AXValueGetValue(value, AXValueType.cgPoint, &point)
return point
}
func setPosition(to: CGPoint) {
var newPoint = to
let position:CFTypeRef? = AXValueCreate(AXValueType(rawValue: kAXValueCGPointType)!, &newPoint)
// What I think is causing issues
let err = AXUIElementSetAttributeValue(windowElement!, kAXPositionAttribute as CFString, position!)
if err != .success {
// I've never seen this happen
print("AXError moving window \(err)")
}
}
}
// CODE THAT GETS EXECUTED ON RECEIVING MOUSE EVENTS
// Gets set to true when in "move mode"
var listeningAndEscapeFlag = false
// I have a notification center observer that updates this when applications are activated, will probably change how this is set eventually
var activePid:pid_t
var transformer:WindowTransformer?
func mouseEventAction(event: CGEvent) -> Unmanaged<CGEvent>? {
let unmodifiedEvent = Unmanaged.passRetained(event)
if !listeningEscapeAndMouseFlag { return unmodifiedEvent }
guard let activePid = activePid else { return unmodifiedEvent }
if transformer == nil {
transformer = WindowTransformer(forWindowWithPid: activePid)
}
let eventLocation = event.location
let deltaEvent = NSEvent.init(cgEvent: event)
let deltaX = deltaEvent?.deltaX
let deltaY = deltaEvent?.deltaY
guard let deltaX = deltaX, let deltaY = deltaY else { return nil }
// Attempt to move window based on mouse events
transformer?.transformWindowWithDeltas(x: deltaX, y: deltaY)
// Keeps cursor visibly frozen
CGWarpMouseCursorPosition(eventLocation)
return nil
}
// Gets passed as callback argument to CGEvent.tapCreate
func mouse_interceptor_callback(tapProxy: CGEventTapProxy,
eventType: CGEventType,
event: CGEvent,
data: UnsafeMutableRawPointer?) -> Unmanaged<CGEvent>? {
// I'm pretty sure modifying mouseMoved events directly doesn't actually do anything
let unmodifiedEvent = Unmanaged.passRetained(event)
if eventType != .mouseMoved {
return unmodifiedEvent
}
return mouseEventAction(event)
}
// HOW I'M CREATING A TAP
var port:CFMachPort?
func createMouseTap() {
let mask:CGEventMask = CGEventMask(1 << CGEventType.mouseMoved.rawValue)
port = CGEvent.tapCreate(tap: CGEventTapLocation.cghidEventTap, // Tap at place where system events enter window server
place: CGEventTapPlacement.headInsertEventTap, // Insert before other taps
options: CGEventTapOptions.defaultTap, // Can modify events
eventsOfInterest: mask,
callback: mouse_interceptor_callback, // fn to run on event
userInfo: nil)
}
func activateTap() {
guard let port = port else { return }
CGEvent.tapEnable(tap: port, enable: true)
let runLoopSrc = CFMachPortCreateRunLoopSource(kCFAllocatorDefault, port, 0)
CFRunLoopAddSource(CFRunLoopGetCurrent(), runLoopSrc, .commonModes)
}
Notes:
- There are some modifications and omissions from my actual code for brevity but this is the code that is directly involved in what I'm attempting to do.
- My computer performs well and there are no delays with other applications (or dragging windows manually) when using multiple displays.
