How to perform Other app list item click using accessibility service like Voice Access app?

Question

I am developing an application like VoiceAccess app. Using accessibility service I am able to perform all clicks which are on top activity(3rd party applications). But I am facing issue with ListItem clicks. I am trying this code for FaceBook app. below is my code. Can any one help me on this.

public class MyService extends AccessibilityService {

/**/
private SharedPreferences S_PREF;
private SharedPreferences.Editor editor;

private static final String TAG = MyService.class
        .getSimpleName();

@Override
public void onAccessibilityEvent(AccessibilityEvent event) {
    clickPerform(getRootInActiveWindow(), 0);
}

public void clickPerform(AccessibilityNodeInfo nodeInfo, final int depth) {

    if (nodeInfo == null) return;

    List<AccessibilityNodeInfo> list = nodeInfo
            .findAccessibilityNodeInfosByViewId("com.facebook.katana:id/bookmarks_tab");
    for (AccessibilityNodeInfo node : list) {

        Log.i(TAG, "ViewID-: bookmarks_tab " + node.getChild(0));

        if (S_PREF.getBoolean("fb_menu", false)) {
            editor.putBoolean("fb_menu", false);
            node.performAction(AccessibilityNodeInfo.ACTION_CLICK);
        }
    }
    List<AccessibilityNodeInfo> list2 = nodeInfo
            .findAccessibilityNodeInfosByViewId("com.facebook.katana:id/bookmarks_list");
    for (AccessibilityNodeInfo node2 : list2) {
        if (node2.getChild(0) != null)
        if (S_PREF.getBoolean("fb_scroll_down", false)) {
            editor.putBoolean("fb_scroll_down", false);
            node2.performAction(AccessibilityNodeInfo.ACTION_SCROLL_FORWARD);
        }
    }
    editor.commit();
    for (int i = 0; i < nodeInfo.getChildCount(); ++i) {
        clickPerform(nodeInfo.getChild(i), depth+1);
    }
}



@Override
public void onInterrupt() {

}

protected void onServiceConnected() {
    S_PREF = getSharedPreferences("S_PREF", Context.MODE_PRIVATE);
    editor = S_PREF.edit();
    AccessibilityServiceInfo info = new AccessibilityServiceInfo();
    info.flags = AccessibilityServiceInfo.DEFAULT;
    info.eventTypes = AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED | AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED;
    info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
    setServiceInfo(info);
    Toast.makeText(getApplicationContext(), "onServiceConnected", Toast.LENGTH_SHORT).show();
     }
}

Below is the VoiceAccess screen. When user says any number, then particular item click will be performed.

I am able to get 7,8,9,10 events but from 11 onwards I am not getting list items individually. My code returning listview id only.

Thanks in advance....

"_But I am facing issue with ListItem clicks_" At a minimum, you need to explain what the "issue" is... i.e. **(a)** exactly what you do (I would _guess_ say a number between 11 and 20, but be explicit); **(b)** exactly what you would expect/hope it would do; and **(c)** exactly what it _does_ do (and if that's "nothing", say so explicitly). — TripeHound, Aug 31 '17 at 13:12
issue is, I am able to get listview id but not list item id/list size etc. If we want to perform item click, we have to know list item position or exact view. So that only we can perform selected item click. I am not understanding how to get that list item view. in above image, if we say 11 that particular item click like instagram opens in browser etc... Like this i have to perform programmatically... thanks for response... — BHARADWAJ Marripally, Aug 31 '17 at 14:05
have you found a solution for this? I'm facing the same issue — steveen zoleko, Dec 25 '17 at 23:31

score 5 · Accepted Answer · answered Dec 26 '17 at 06:40

@steveenzoleko

I got solution for this problem by myself. Here AccessibilityNodeInfo always doesn't return complete list or list size. It returns VIEWs. It may be TextView, Button etc... Every View has multiple methods like getText(), getContentDescription(), getClassName(), getChildCount(), getViewIdResourceName() etc... here Voice Access app detecting all views and giving them some numbers. For lists, using getChildCount() method in for loop we can getChildViews/listItems. Ex:

AccessibilityNodeInfo node = getRootInActiveWindow();
if(node != null) {
   for(int i = 0; i < node.getChildCount(); i++){
      AccessibilityNodeInfo childNode = node.getChild(i);
      if(childNode != null){
        Log.i("childNode", "-----getText->"+childNode.getText()+"---getContentDescription-->"+childNode.getContentDescription() );
      }
   }
}

use getText() and getContentDescription() methods to know text of textview, button, checkbox.

getClassName() method retuns the view type like TextView, Button, ImageView, CheckBox etc...

Can you answer this question https://stackoverflow.com/q/49734263/4813904 — Siva, Apr 09 '18 at 15:48
@BHARADWAJ Marripally: I have the same problem as you do, with the items from certain listviews. Unfortunately, I can't send the click with `childNode.performAction(AccessibilityNodeInfo.ACTION_CLICK)` even if I try to send the click to the parent view, I can't because `childNode.isClickable()` return false. Can you help me with a snippet or link idea? — dole doug, Nov 04 '19 at 16:04

score 0 · Answer 2 · answered Aug 27 '20 at 11:56

0

Maybe you can use dispatchGesture from AccessibilityService API 7.0.

answered Aug 27 '20 at 11:56

ekse

1
5

How to perform Other app list item click using accessibility service like Voice Access app?

2 Answers2

Linked