How to trick an OS X app into thinking the mouse is a finger?

Question

I'm writing a Mac app that contains a collection view. This app is to be run on a large touchscreen display (55" EP series from Planar). Due to hardware limitation, the touchscreen doesn't send scroll events (or even any multitouch events). How can I go about tricking the app into thinking a "mousedown+drag" is the same as a "mousescroll"?

I got it working halfway by subclassing NSCollectionView and implementing my own NSPanGestureRecognizer handler in it. Unfortunately the result is clunky and doesn't have the feeling of a normal OS X scroll (i.e., the velocity effect at the end of a scroll, or scroll bounce at the ends of the content).

@implementation UCTouchScrollCollectionView
...
- (IBAction)showGestureForScrollGestureRecognizer:(NSPanGestureRecognizer *)recognizer
{
    CGPoint location = [recognizer locationInView:self];

    if (recognizer.state == NSGestureRecognizerStateBegan) {

        touchStartPt = location;
        startOrigin = [(NSClipView*)[self superview] documentVisibleRect].origin;

    } else if (recognizer.state == NSGestureRecognizerStateEnded) {

        /* Some notes here about a future feature: the Scroll Bounce
           I don't want to have to reinvent the wheel here, but it
           appears I already am. Crud.

           1. when the touch ends, get the velocity in view
           2. Using the velocity and a constant "deceleration" factor, you can determine
               a. The time taken to decelerate to 0 velocity
               b. the distance travelled in that time
           3. If the final scroll point is out of bounds, update it.
           4. set up an animation block to scroll the document to that point. Make sure it uses the proper easing to feel "natural".
           5. make sure you retain a pointer or something to that animation so that a touch DURING the animation will cancel it (is this even possible?)
        */

        [self.scrollDelegate.pointSmoother clearPoints];
        refreshDelegateTriggered = NO;

    } else  if (recognizer.state == NSGestureRecognizerStateChanged) {

        CGFloat dx = 0;
        CGFloat dy = (startOrigin.y - self.scrollDelegate.scrollScaling * (location.y - touchStartPt.y));
        NSPoint scrollPt = NSMakePoint(dx, dy);

        [self.scrollDelegate.pointSmoother addPoint:scrollPt];
        NSPoint smoothedPoint = [self.scrollDelegate.pointSmoother getSmoothedPoint];
        [self scrollPoint:smoothedPoint];

        CGFloat end = self.frame.size.height - self.superview.frame.size.height;
        CGFloat threshold = self.superview.frame.size.height * kUCPullToRefreshScreenFactor;
        if (smoothedPoint.y + threshold >= end &&
            !refreshDelegateTriggered) {
            NSLog(@"trigger pull to refresh");
            refreshDelegateTriggered = YES;
            [self.refreshDelegate scrollViewReachedBottom:self];
        }
    }
}

A note about this implementation: I put together scrollScaling and pointSmoother to try and improve the scroll UX. The touchscreen I'm using is IR-based and gets very jittery (especially when the sun is out).

In case it's relevant: I'm using Xcode 6 beta 6 (6A280e) on Yosemite beta (14A329r), and my build target is 10.10.

Thanks!

The OS X multitouch API doesn't support event injection unless you do some *extremely* dirty stuff (building internal event structs by hand and dropping them into the HID event stream...and even that doesn't always work). I've been bitten by this a number of times - I'd love to see an answer to this question. — nneonneo, Aug 26 '14 at 22:43
Years ago, I did something like this by generating multitouch events with Cocoa, converting them to CGEvent, then converting to Carbon events (which required reverse engineering how touch events were represented in Carbon, because normally they don't show up in the Carbon stream at all…), then pushing them into the Carbon event stream. Unfortunately, trying to compile that code for 64-bit even on an older version of OS X gives me a whole slew of errors, so I suspect that no longer works. — abarnert, Aug 26 '14 at 23:16

score 1 · Answer 1 · answered Mar 24 '16 at 23:04

I managed to have some success using an NSPanGestureRecognizer and simulating the track-pad scroll wheel events. If you simulate them well you'll get the bounce from the NSScrollView 'for free'.

I don't have public code, but the best resource I found that explained what the NSScrollView expects is in the following unit test simulating a momentum scroll. (See mouseScrollByWithWheelAndMomentumPhases here).

https://github.com/WebKit/webkit/blob/master/LayoutTests/fast/scrolling/latching/scroll-iframe-in-overflow.html

The implementation of mouseScrollByWithWheelAndMomentumPhases gives some tips on how to synthesize the scroll events at a low level. One addition I found I needed was to actually set an incrementing timestamp in the event in order to get the scroll-view to play ball.

https://github.com/WebKit/webkit/blob/master/Tools/WebKitTestRunner/mac/EventSenderProxy.mm

Finally, in order to actually create the decaying velocity, I used a POPDecayAnimation and tweaked the velocity from the NSPanGestureRecognizer to feel similar. Its not perfect but it does stay true to NSScrollView's bounce.

score 0 · Answer 2 · edited May 23 '17 at 12:07

I have a (dead) project on Github that does this with an NSTableView, so hopefully it will work well for an NSCollectionView.

Disclaimer: I wrote this while I was still learning GCD, so watch for retain cycles... I did not vet what I just posted for bugs. feel free to point any out :) I just tested this on Mac OS 10.9 and it does still work (originally written for 10.7 IIRC), not tested on 10.10.

This entire thing is a hack to be sure, it looks like it requires (seems to anyway) asynchronous UI manipulation (I think to prevent infinite recursion). There is probably a cleaner/better way and please share it when you discover it!

I havent touched this in months so I cant recall all the specifics, but the meat of it surely is in the NBBTableView code, which will paste snippets of.

first there is an NSAnimation subclass NBBScrollAnimation that handles the "rubber band" effect:

@implementation NBBScrollAnimation

@synthesize clipView;
@synthesize originPoint;
@synthesize targetPoint;

+ (NBBScrollAnimation*)scrollAnimationWithClipView:(NSClipView *)clipView
{
    NBBScrollAnimation *animation = [[NBBScrollAnimation alloc] initWithDuration:0.6 animationCurve:NSAnimationEaseOut];

    animation.clipView = clipView;
    animation.originPoint = clipView.documentVisibleRect.origin;
    animation.targetPoint = animation.originPoint;

    return [animation autorelease];
}

- (void)setCurrentProgress:(NSAnimationProgress)progress
{
    typedef float (^MyAnimationCurveBlock)(float, float, float);
    MyAnimationCurveBlock cubicEaseOut = ^ float (float t, float start, float end) {
        t--;
        return end*(t * t * t + 1) + start;
    };

    dispatch_sync(dispatch_get_main_queue(), ^{
        NSPoint progressPoint = self.originPoint;
        progressPoint.x += cubicEaseOut(progress, 0, self.targetPoint.x - self.originPoint.x);
        progressPoint.y += cubicEaseOut(progress, 0, self.targetPoint.y - self.originPoint.y);

        NSPoint constraint = [self.clipView constrainScrollPoint:progressPoint];
        if (!NSEqualPoints(constraint, progressPoint)) {
            // constraining the point and reassigning to target gives us the "rubber band" effect
            self.targetPoint = constraint;
        }

        [self.clipView scrollToPoint:progressPoint];
        [self.clipView.enclosingScrollView reflectScrolledClipView:self.clipView];
        [self.clipView.enclosingScrollView displayIfNeeded];
    });
}

@end

You should be able to use the animation on any control that has an NSClipView by setting it up like this _scrollAnimation = [[NBBScrollAnimation scrollAnimationWithClipView:(NSClipView*)[self superview]] retain];

The trick here is that the superview of an NSTableView is an NSClipView; I dont know about NSCollectionView, but I suspect that any scrollable control uses NSClipView.

Next here is how the NBBTableView subclass makes use of that animation though the mouse events:

- (void)mouseDown:(NSEvent *)theEvent
{
    _scrollDelta = 0.0;
    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
        if (_scrollAnimation && _scrollAnimation.isAnimating) {
            [_scrollAnimation stopAnimation];
        }
    });
}

- (void)mouseUp:(NSEvent *)theEvent
{
    if (_scrollDelta) {
        [super mouseUp:theEvent];
        // reset the scroll animation
        dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
            NSClipView* cv = (NSClipView*)[self superview];
            NSPoint newPoint = NSMakePoint(0.0, ([cv documentVisibleRect].origin.y - _scrollDelta));

            NBBScrollAnimation* anim = (NBBScrollAnimation*)_scrollAnimation;
            [anim setCurrentProgress:0.0];
            anim.targetPoint = newPoint;

            [anim startAnimation];
        });
    } else {
        [super mouseDown:theEvent];
    }
}

- (void)mouseDragged:(NSEvent *)theEvent
{
    NSClipView* clipView=(NSClipView*)[self superview];
    NSPoint newPoint = NSMakePoint(0.0, ([clipView documentVisibleRect].origin.y - [theEvent deltaY]));
    CGFloat limit = self.frame.size.height;

    if (newPoint.y >= limit) {
        newPoint.y = limit - 1.0;
    } else if (newPoint.y <= limit * -1) {
        newPoint.y = (limit * -1) + 1;
    }
    // do NOT constrain the point here. we want to "rubber band"
    [clipView scrollToPoint:newPoint];
    [[self enclosingScrollView] reflectScrolledClipView:clipView];

    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0), ^{
        NBBScrollAnimation* anim = (NBBScrollAnimation*)_scrollAnimation;
        anim.originPoint = newPoint;
    });

    // because we have to animate asyncronously, we must save the target value to use later
    // instead of setting it in the animation here
    _scrollDelta = [theEvent deltaY] * 3.5;
}

- (BOOL)autoscroll:(NSEvent *)theEvent
{
    return NO;
}

I think that autoscroll override is essential for good behavior.

The entire code is on my github page, and it contains several other "touch screen" emulation tidbits, if you are interested, such as a simulation for the iOS springboard arrangeable icons (complete with "wiggle" animation using NSButtons.

Hope this helps :)

Edit: It appears that constrainScrollPoint: is deprecated in OS X 10.9. However, It should fairly trivial to reimplement as a category or something. Maybe you can adapt a solution from this SO question.

thanks! I'm just starting to poke through this. I notice you have a couple of classes NBBVirtualKeyboard and NBBKeyboardKeyCell in there. Were those abandoned in favor of a different keyboard solution? They don't seem to be implemented. — Spencer Williams, Sep 04 '14 at 19:07
@SpencerWilliams This code is incomplete. it is sort of a rewrite of an earlier project that *did* have a virtual keyboard. I do still have that code if it would benefit you at all. — Brad Allred, Sep 04 '14 at 19:16
yeah, any keyboard code you have would be a huge help! We have to implement an onscreen keyboard ourselves, and it's a pretty daunting task — Spencer Williams, Sep 04 '14 at 19:20
I've started an open source "soft keyboard" library [here](https://github.com/spilliams/SWSoftKeyboard). — Spencer Williams, Sep 05 '14 at 19:13

How to trick an OS X app into thinking the mouse is a finger?

2 Answers2