How can I use AccessibilityService.getWindows() to obtain a traversable AccessibilityNodeInfo?

Question

I am writing an AccessibilityService for Android and, up to API level 20, I have been using the AccessibilityEvent.getSource() method to obtain a traversable AccessibilityNodeInfo when onAccessibilityEvent(AccessibilityEvent event) is triggered. Although the resulting AccessibilityNodeInfo does not always reflect the content of the screen, it is still possible to work with it.

Starting on API level 21, the new AccessibilityService.getWindows() is supposed to be not only able to better represent the view hierarchy (i.e., Z-ordering is respected), but it is also supposed to be able to expose a node that includes all views in the current input method (IME). I would like to take advantage of this but I have not been able to do so and I do not know what exactly I am doing wrong. Incidentally, I have been unable to find any more detailed information on how to do this, other than the very minimal java docs.

I have already done the following:

Configured service to retrieve window content (android:canRetrieveWindowContent="true")
Added flagRetrieveInteractiveWindows to service flags

My code is as follows:

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
            ArrayList<AccessibilityNodeInfo> nodes = getNodesFromWindows();
                switch (event_type) {
                case AccessibilityEvent.TYPE_VIEW_ACCESSIBILITY_FOCUSED:
                case AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED:
                case AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED:
                //case AccessibilityEvent.TYPE_VIEW_TEXT_CHANGED:
                case AccessibilityEvent.TYPE_VIEW_FOCUSED:
                case AccessibilityEvent.TYPE_VIEW_SELECTED:
                case AccessibilityEvent.TYPE_VIEW_SCROLLED:
                //case AccessibilityEvent.TYPE_VIEW_CLICKED:
                    updateTargetLeafs(nodes);
                }
}

where getNodesFromWindows() does the following:

private ArrayList<AccessibilityNodeInfo> getNodesFromWindows() {
    List<AccessibilityWindowInfo> windows = getWindows();
    ArrayList<AccessibilityNodeInfo> nodes =
            new ArrayList<AccessibilityNodeInfo>();
    if (windows.size() > 0) {
        for (AccessibilityWindowInfo window : windows) {
            nodes.add(window.getRoot());
        }
    }
    return nodes;
}

after this, updateTargetLeafs() collects all clickable, enabled and visible nodes into a separate AccessibilityNodeInfo ArrayList so I can index and access them at will (see below). When using AccessibilityEvent.getSource() on API Level 20 and below, the size of this array is always close to the number of views on the screen, but when I use AccessibilityService.getWindows() the size is almost always 1 (sometimes 0), and the bounds of the only AccessibilityNodeInfo in the list are always outside of the screen.

EDIT: Add the code for iterating through all nodes children (where mNodes is the output of getNodesFromWindows()):

        ...
        ArrayList<AccessibilityNodeInfo> theseleafs =
                    new ArrayList<AccessibilityNodeInfo>();
        AccessibilityNodeInfo thisnode;
        Queue<AccessibilityNodeInfo> q =
                new LinkedList<AccessibilityNodeInfo>();
        for (AccessibilityNodeInfo n : mNodes) {
            q.add(n);
        }
        while (!q.isEmpty()) {
            thisnode = q.poll();
            if (shouldIncludeNode(thisnode)) {
                //Add only if it fulfills all requirements!
                theseleafs.add(thisnode);
            }
            for (i=0; i<thisnode.getChildCount(); ++i) {
                AccessibilityNodeInfo n = thisnode.getChild(i);
                if (n != null) q.add(n); // Add only if not null!
            }
        };
        LogD("MyTag", theseleafs.size() + " leafs in this node!");
        ...

Odd, I know, but what am I doing wrong?

score 5 · Answer 1 · answered May 05 '15 at 03:25

5

You can get access of windows content using getChild() method. In your onAccessibilityEvent(AccessibilityEvent event) you can do like below.

@Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
          AccessibilityNodeInfo mSource = event.getSource();
          int child = mSource.getChildCount();
          // iterate through all child of parent view
          for (int i=0; i<child; i++){
            AccessibilityNodeInfo childNodeView = mParent.getChild(i);
            // Do something with this window content
          }
}

answered May 05 '15 at 03:25

Vijay Vankhede

3,018
1
26
46

i have one doubt. we are getting all child node thats ok but how to get the source of all child nodes? Is this Possible? through the event we can get source of node which is triggered by some event. for child nodes how we can get source? @vsvankahede – Happy Jun 18 '15 at 06:19
1

First get parent of source node by getParent(). Then find number of the child of parent by getChildCount(). Use for loop and iterate through all child and store there reference to ArrayList. Now you can do what ever you want with this accessibility node – Vijay Vankhede Jun 18 '15 at 09:32
yeah but how to get the source of child nodes? – Happy Jun 18 '15 at 10:37
Please read this http://developer.android.com/reference/android/view/accessibility/AccessibilityRecord.html#getSource() – Vijay Vankhede Jun 24 '15 at 10:55
1

what do you mean by 'source' of child nodes? In `AccessibilityService` you receive `AccessibilityEvent`s and when you call `getSource()` on one event, you get an `AccessibilityNodeInfo` object. So, the node is the source, means the source of the event. – Prasad Pawar Dec 14 '15 at 06:57
@vsvankhede I specifically mentioned that I am currently using event.getSource() and that it is working. This does not answer the question at all, which is... how to obtain the accessibility information using the new getWindows() method available in API Level 21 and above. – J-Doc Mar 23 '16 at 18:00

score 0 · Answer 2 · answered Feb 23 '15 at 16:58

0

getNodesFromWindows() is only returning a List of root nodes. You need to iterate through its children to collect all nodes in the window.

answered Feb 23 '15 at 16:58

Phil Weaver

738
3
7

yup, that is exactly what updateTargetLeafs() does. I have updated the question to show this. – J-Doc Feb 23 '15 at 17:25

How can I use AccessibilityService.getWindows() to obtain a traversable AccessibilityNodeInfo?

2 Answers2

Linked