Retrieve TYPE_TOUCH and TYPE_GESTURE in AccessibilityService

Question

I am developing an application which is supposed to collect user's interaction with Android device (touch related data) similar to this SO post.

I've tried all of the answer's suggested approaches starting from an app overlay with and without WATCH_OUTSIDE_TOUCH flag. getevent approach is also not an option for me as it requires root or external storage (as the answer states).

So I've created an AccessibilityService and now I am able to detect such events as: VIEW_CLICKED, VIEW_SCROLLED. However in the documentation I've encountered also GESTURE_DETECTION_START, GESTURE_DETECTION_END, TOUCH_INTERACTION_STARTand TOUCH_INTERACTION_END. I would like to use them to measure the interaction time. Unfortunately my service is not retrieving this events at all despite proper (?) configuration. My current configuration looks like:

AndroidManifest.xml:

...
<service
    android:name=".accessibility.MyAccessibilityService"
    android:label="@string/accessibility_service_label"
    android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
    <intent-filter>
        <action android:name="android.accessibilityservice.AccessibilityService" />
    </intent-filter>
    <meta-data
        android:name="android.accessibilityervice"
        android:resource="@xml/accessibility_config" />
</service>
...

accessibility_config.xml:

<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:description="@string/accessibility_service_description"
    android:notificationTimeout="5"
    android:summary="@string/accessibility_service_description" />

Summing up, my service is starting and retrieving some events properly, but it lacks gesture and touch start/end events. I've tried many other configurations including all capabilities and flags enabled but without any effect. Also I tried overriding a method AccessibilityService::onGesture but it's never called. Are this events available only in touch exploration mode?

I've described my problem maybe a little too broad, but I would appreciate any help or suggestions even not related strictly to AccessibilityService approach.

Jakub Licznerski · Accepted Answer · 2021-05-14T15:26:25.290

2

After some struggling I found out what was the problem (or set of problems):

My accessibility_config.xml file was invalid because it contained <?xml version="1.0" encoding="utf-8"?> tag, and according to the docs

This meta-data must reference an XML resource containing < accessibility-service > tag

So I deducted that it needs this and only this tag to work correctly.

GESTURE_DETECTION_START, GESTURE_DETECTION_END, TOUCH_INTERACTION_START and TOUCH_INTERACTION_END are events available only in touch exploration mode.

So android:canRequestTouchExplorationMode="true" and android:accessibilityFlags="flagRequestTouchExplorationMode" were missing in my config. The docs enlist those events under Exploration types.

EDIT: sharing the final accessibility_settings.xml file:

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeViewTextChanged|typeViewFocused|typeViewLongClicked|typeViewClicked|typeViewScrolled|typeWindowStateChanged"
    android:accessibilityFeedbackType="feedbackHaptic"
    android:accessibilityFlags="flagIncludeNotImportantViews"
    android:canRetrieveWindowContent="true"
    android:description="@string/accessibility_service_description"
    android:notificationTimeout="10"
    android:settingsActivity="org.example.ExampleActivity" />

edited May 14 '21 at 15:26

answered Jun 05 '18 at 19:32

Jakub Licznerski

1,008
1
17
33

Does it allow to be notified of all touch events ? Can you please share a Github sample of this? – android developer Dec 14 '20 at 19:10
Hi @androiddeveloper, sorry for the late reply! I edited my post with final xml file that I used in the project. It allows listening for all of the declared events. The project is proprietary so I can't share the whole repository, but hit me up if you have any further questions. I develop it a few years back, so my memory is a bit blurry in the details, however I remember that the mask `typeAllMask` did not work in some cases so I used explicit types mask. – Jakub Licznerski May 14 '21 at 15:30
So this can work when used on any app, including even games? It can record what the user is doing? Is it also possible to do the opposite? Meaning to perform the touch events? This way, you could record macros... – android developer May 30 '21 at 19:27
Yes, that's correct. As long as the game view is managed by your app. The exact touch coordinates do not work e.g. in overlays. The other downside would be that it needs specific accessibility permissions granted manually by the user (you can have an intent pointing to appropriate settings page, but not just a prompt to give permissins). I haven't tried the other direction, but I suspect it might be possible. Good luck :) – Jakub Licznerski Jun 01 '21 at 08:14
What do you mean by "As long as the game view is managed by your app" ? – android developer Jun 01 '21 at 15:47
I meant not using an overlay view over some other application (i.e. game). In this case you can intercept a touch event, but can't pass it to other views, down. That was my research, trying to build basically a keylogger for phone (for non-criminal use cases ofc :D) – Jakub Licznerski Jun 02 '21 at 16:27
If your app has an overlay view, you can't mimic touch events? Not sure I understand. Do you have a POC for this? – android developer Jun 02 '21 at 20:18
I was trying to capture all touch events (especially those outside my application) to track user's interaction with the smartphone. Tried with an overlay view (always on top, invisible) that would act as an interceptor for those events. Idea was not to consume the touch event and pass it down to the actual application view, but due to security reasons that is not possible. I did try to programatically trigger the same touch event but I did not succeed and unfortunately don't have the code for that anymore. – Jakub Licznerski Jun 04 '21 at 10:18
I see, so this is not related to accessibility, right? – android developer Jun 05 '21 at 08:27
yes, rather using android's accessibility features for interaction tracking – Jakub Licznerski Jun 06 '21 at 09:26
Wow cool. I wonder if there are apps that use it. Could be great to have macro recording and playing. – android developer Jun 06 '21 at 11:29
@androiddeveloper I must have been brainfogged when answering your question. The touch events emmited from accessibility are not regular events, you don't have access to exact coordinates or view elements that are being clicked. You can however track occurrences of click, long click, scroll and focus change (also text changes from not sensitive input views). This can be done system-wide, no need for an overlay. Let me know if you still need it, I might be able to share some code bits – Jakub Licznerski Oct 29 '21 at 00:21
Are you sure ? There are some auto-clicker apps and they seem to require accessibility... – android developer Oct 29 '21 at 01:24
Performing click is another story, however still not trivial. You'd need to know the other app's layout to click on a specific element: https://stackoverflow.com/questions/26628262/how-to-click-button-in-settings-using-accessibilityservice. As of retrieving the only dimension you get from touch events is scrollX and Y – Jakub Licznerski Oct 29 '21 at 06:56
I saw auto-clicker apps for games, and those rarely use the layout system of the OS. – android developer Oct 31 '21 at 08:54

Retrieve TYPE_TOUCH and TYPE_GESTURE in AccessibilityService

1 Answers1