0

I'm coding a macOS app with Xcode (XCode 11.5 under MacOS Catalina), the goal being to send through UDP the coordinates of fingers in contact with my Macbook Pro's trackpad, and trigger an haptic feedback when a UDP message is received.

Being a novice in Xcode, I created my app with this code, and SwiftOSC for the UDP communication. I succeeded to do what I wanted when the app is in the foreground.

Code: exactly the one found here.

Problem: the app is not updated while being in the background.

Update: Following @zrzka advices:

  1. I use
NSEvent.addGlobalMonitorForEvents(matching: NSEvent.EventTypeMask.any) {event in
    self.backMouseMovements(with: event)
}

to monitor events in the background, with the following function:

func backMouseMovements(with event: NSEvent) {
    print("Event: \(event)")
    let touches = event.touches(matching: .moved, in: self)
    delegate?.touchesView(self, didUpdateTouchingTouches: touches)
}

I put this code in override init(frame frameRect: NSRect) of the AppKitTouchesView class of this code.

  1. Accessibility: AXIsProcessTrusted() returns true

  2. I don't have access to touches with these events, and even less to normalizedPosition, which is my ultimate goal.

Instead, I have an error:

2020-08-13 09:40:53.538119+0200 TrackPad[1263:34805] *** Assertion failure in -[NSEvent touchesMatchingPhase:inView:], /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1894.50.103/AppKit.subproj/NSEvent.m:4874
2020-08-13 09:40:53.538198+0200 TrackPad[1263:34805] [General] An uncaught exception was raised
2020-08-13 09:40:53.538229+0200 TrackPad[1263:34805] [General] Invalid message sent to event "NSEvent: type=MouseMoved loc=(1197.92,994.461) time=3138.3 flags=0 win=0x0 winNum=150 ctxt=0x0 evNum=307 click=1 buttonNumber=0 pressure=0 deltaX=1.000000 deltaY=0.000000 deviceID:0x300000080500000 subtype=NSEventSubtypeTouch"
2020-08-13 09:40:53.539294+0200 TrackPad[1263:34805] [General] (
    0   CoreFoundation                      0x00007fff318c5be7 __exceptionPreprocess + 250
    1   libobjc.A.dylib                     0x00007fff6a69d5bf objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff318eed98 +[NSException raise:format:arguments:] + 88
    3   Foundation                          0x00007fff33fdde9d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 191
    4   AppKit                              0x00007fff2ee7fc36 -[NSEvent touchesMatchingPhase:inView:] + 449
    5   TrackPad                            0x0000000100002d0d $s8TrackPad17AppKitTouchesViewC18backMouseMovements4withySo7NSEventC_tF + 813
    6   TrackPad                            0x0000000100002986 $s8TrackPad17AppKitTouchesViewC5frameACSo6CGRectV_tcfcySo7NSEventCcfU_ + 214
    7   TrackPad                            0x0000000100002f09 $sSo7NSEventCIegg_ABIeyBy_TR + 73
    8   AppKit                              0x00007fff2f0a14e9 GlobalObserverHandler + 89
    9   HIToolbox                           0x00007fff3044f8ff _ZL23DispatchEventToHandlersP14EventTargetRecP14OpaqueEventRefP14HandlerCallRec + 1254
    10  HIToolbox                           0x00007fff3044ed8d _ZL30SendEventToEventTargetInternalP14OpaqueEventRefP20OpaqueEventTargetRefP14HandlerCallRec + 329
    11  HIToolbox                           0x00007fff3044ec3d SendEventToEventTargetWithOptions + 45
    12  HIToolbox                           0x00007fff3048eb8c _ZL29ToolboxEventDispatcherHandlerP25OpaqueEventHandlerCallRefP14OpaqueEventRefPv + 1329
    13  HIToolbox                           0x00007fff3044fe8a _ZL23DispatchEventToHandlersP14EventTargetRecP14OpaqueEventRefP14HandlerCallRec + 2673
    14  HIToolbox                           0x00007fff3044ed8d _ZL30SendEventToEventTargetInternalP14OpaqueEventRefP20OpaqueEventTargetRefP14HandlerCallRec + 329
    15  HIToolbox                           0x00007fff3046447e SendEventToEventTarget + 39
    16  AppKit                              0x00007fff2eabe9aa _DPSNextEvent + 1268
    17  AppKit                              0x00007fff2eabd070 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1352
    18  AppKit                              0x00007fff2eaaed7e -[NSApplication run] + 658
    19  AppKit                              0x00007fff2ea80b86 NSApplicationMain + 777
    20  TrackPad                            0x000000010000eead main + 13
    21  libdyld.dylib                       0x00007fff6b844cc9 start + 1

I guess this error is thrown because I want to access to touches, which doesn't exist for these events. However, I don't understand how to make a link between the NSEvent of my addGlobalMonitorForEvents and NSTouch, where normalizedPosition is available.

Thanks!

BBrslt
  • 31
  • 2
  • 2
    Welcome! Please, take the [Tour](https://stackoverflow.com/tour) and read [What types of questions should I avoid asking?](https://stackoverflow.com/help/dont-ask), [What topics can I ask about here?](https://stackoverflow.com/help/on-topic) and [How do I ask a good question?](https://stackoverflow.com/help/how-to-ask) -- You just shared what you're doing and a couple of libraries/code you're using. But you should provide code of your app as well, parts which don't work. My guess is that your app is just napping. See [my answer](https://stackoverflow.com/a/61894359/581190). – zrzka Aug 10 '20 at 12:16
  • Actually, the app works pretty well when not being in the background. I didn't know that apps could nap, I'll read about this and your answer, thanks! – BBrslt Aug 10 '20 at 13:23
  • Instead of linking, I would add a brief excerpt of the code in question, and then explain how the code is not performing as expected / feature you want to add. – gannonbarnett Aug 10 '20 at 15:02
  • If I understand you correctly, you'd like to send trackpad finger positions even if the app is in the background? Actually, I wrote the code you linked and it doesn't work in a way you're expecting. It was written to handle view touches. But you need a global event monitor to monitor mouse events even if your app is in the background. Just search SO, plenty of examples around and combine them with my code to calculate finger position. – zrzka Aug 10 '20 at 18:26
  • @zrzka Yep exactly. I thought I just had to add a line of code to send the data through OSC, like this (in the _struct Touch_ of your code): `init(_ nsTouch: NSTouch) { self.normalizedX = nsTouch.normalizedPosition.x self.normalizedY = 1.0 - nsTouch.normalizedPosition.y self.id = nsTouch.hash self.position = [] client.send(OSCMessage(OSCAddressPattern("/position"), Float(normalizedX), Float(normalizedY))) }` Thanks for your advices! As a novice, I found it hard to understand how this works, thank you to have taken the time to answer :) – BBrslt Aug 11 '20 at 09:26
  • Couple of places where to start ... 1) Global event monitor - [`addGlobalMonitorForEvents(matching:handler:)`](https://developer.apple.com/documentation/appkit/nsevent/1535472-addglobalmonitorforevents) 2) [Accessibility](https://stackoverflow.com/a/61890478/581190) to be able to globally monitor events 3) [`touches(matching:in:)`](https://developer.apple.com/documentation/appkit/nsevent/1530950-touches) to get touches (hopefully with `normalizedPosition`). All pointers you need to implement this. Try it and update your question if something wont work for you. – zrzka Aug 11 '20 at 09:40
  • I have access to the position of the mouse in the window, but I found no way to obtain the finger's position on the trackpad... Any idea? – BBrslt Aug 13 '20 at 15:13

1 Answers1

0

I finally found a solution with this framework: https://github.com/Kyome22/OpenMultitouchSupport

Thanks for your help!

BBrslt
  • 31
  • 2