Full screen TYPE_ACCESSIBILITY_OVERLAY

Question

I'm new to the accessibility stuff on Android. While going through the classes and documentation I came across TYPE_ACCESSIBILITY_OVERLAY inside the WindowManager class.

The documentation says (only the relevant text)

For example, if there is a full screen accessibility overlay that is touchable, the windows below it will be introspectable by an accessibility service even though they are covered by a touchable window.

So I set out to achieve just that, a full screen accessibility overlay and try to introspect the windows below it

Extended AccessibilityService and added my full screen overlay when onServiceConnected is called (the inspiration for adding overlay came from here)

@Override
protected void onServiceConnected() {
    WindowManager wm = (WindowManager) getSystemService(WINDOW_SERVICE);
    FrameLayout mLayout = new FrameLayout(this);
    WindowManager.LayoutParams lp = new WindowManager.LayoutParams();
    lp.type = WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY;
    lp.format = PixelFormat.TRANSLUCENT;
    lp.flags |= WindowManager.LayoutParams.FLAG_FULLSCREEN;
    lp.width = WindowManager.LayoutParams.MATCH_PARENT;
    lp.height = WindowManager.LayoutParams.MATCH_PARENT;
    lp.gravity = Gravity.TOP;
    wm.addView(mLayout, lp);

    mLayout.setOnTouchListener(new View.OnTouchListener() {
        @Override
        public boolean onTouch(View v, MotionEvent event) {
            // Here I'm getting the touch events on the overlay I added
            return false;
        }
    });
}

Now, the question is, how do I introspect or find the windows below this overlay? Even in the onAccessibilityEvent callback I get just this overlay window. getWindows() always has a size of 1. Doesn't it refute the assertion made above for TYPE_ACCESSIBILITY_OVERLAY?

Relevant info: To receive the touch events on the overlay I have disabled touchExplorationMode in the service settings

android:canRequestTouchExplorationMode="false"

randomuser · Accepted Answer · 2018-06-14T15:04:30.690

What you seem to be missing is flagRetrieveInteractiveWindows on your configuration. These properties and window layout paremeters configuration should work, without requiring for you to disable canRequestTouchExplorationMode in order to get the events and having getWindows return the AccessibilityWindowInfo instances underneath yours:

    <?xml version="1.0" encoding="utf-8"?>
    <accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
        android:packageNames="test.demo.com.tests"
        android:accessibilityEventTypes="typeAllMask"
        android:accessibilityFlags="flagRetrieveInteractiveWindows|flagReportViewIds|flagIncludeNotImportantViews"
        android:accessibilityFeedbackType="feedbackAllMask"
        android:notificationTimeout="100"
        android:canRetrieveWindowContent="true"
        />

And on service connected:

    @Override
    protected void onServiceConnected() {
        WindowManager windowManager = (WindowManager) getSystemService(WINDOW_SERVICE);
        FrameLayout layout = new FrameLayout(this);

        WindowManager.LayoutParams params = new WindowManager.LayoutParams(WindowManager.LayoutParams.MATCH_PARENT,
            WindowManager.LayoutParams.MATCH_PARENT, WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
                WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE| WindowManager.LayoutParams.FLAG_FULLSCREEN |
                WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE|
                WindowManager.LayoutParams.FLAG_DRAWS_SYSTEM_BAR_BACKGROUNDS| 
                WindowManager.LayoutParams.FLAG_WATCH_OUTSIDE_TOUCH,
            PixelFormat.TRANSLUCENT);
        params.gravity = Gravity.TOP;

        windowManager.addView(layout, params);
        layout.setOnTouchListener(new View.OnTouchListener() {
            @Override
            public boolean onTouch(View v, MotionEvent event) {
                //You can either get the information here or on onAccessibilityEvent                 
                return false;
            }
        });
    }

EDIT:

Added FLAG_DRAWS_SYSTEM_BAR_BACKGROUNDS to accomplish full screen and removed canRequestTouchExplorationMode since the flag associated to this property should not be included and, therefore, of no use.

Thanks for your answer @randomuser. I'll check it in my setup and get back to you. — Rajeev, Jun 11 '18 at 12:54
So, I have checked your code against my listed requirement, I find that your code does give me Window size 2, but there are a few caveats to your assertion. 1. Your code is not using `touchExplorationMode` as you haven't added the same in `accessibilityFlags` in your xml settings, 2. Your overlay is not full screen if you include `WindowManager.LayoutParams.FLAG_NOT_TOUCHABLE` (I checked `boundsInParent`). Since #1 was not my requirement, and #2 I was able to find out through trial, I am rewarding you the bounty, but please update your answer after verifying my claims. Thanks — Rajeev, Jun 12 '18 at 06:43
Regarding the first aforementioned caveat, `canRequestTouchExplorationMode` is of no use without `flagRequestTouchExplorationMode` included as part of your accessibility flags. That, added to the fact that the question didn't include the meta data, makes the relevant information section not that pertinent. — randomuser, Jun 14 '18 at 15:11
As a side note, this excerpt from the specs might affect your service in the future (blocking touch events) `The system will enable touch exploration mode if there is at least one accessibility service that has this flag set. Hence, clearing this flag does not guarantee that the device will not be in touch exploration mode since there may be another enabled service that requested it.` — randomuser, Jun 14 '18 at 15:12

Full screen TYPE_ACCESSIBILITY_OVERLAY

1 Answers1