Android Clickable On screen text

Question

I was curious, I seen this app the other day that allowed it to open other apps and set certain functions up for you automatically. I have came to realize that it must be using an on screen click function of some sort, but I can't seem to find any documentation for something like this. For example if we know the on screen text from the other app is "Ready", is there a way to read that text and maybe do something like:

protected void processText(String text)
{
  if (text.contains("Ready"))
      // click the ready text
}

Do you want to click text or make part of a text clickable, or get screen click coordinates? Information on the former could be gotten here: https://www.google.com.ng/url?url=https://stackoverflow.com/questions/10696986/how-to-set-the-part-of-the-text-view-is-clickable&rct=j&sa=U&ved=0ahUKEwilotjX6-7UAhXIYlAKHfF8BwgQFggbMAA&q=android+make+part+of+text+clickable&usg=AFQjCNEVpYcyHrHbO5NKt5jBPVqAjpNiFg — Mofe Ejegi, Jul 04 '17 at 05:01
@Mofe-hendyEjegi I want to click the text without the user having to do it themselves. But since it's on another app, I can't make it click, that is what I am trying to figure out how to do. — Jayce, Jul 04 '17 at 15:13
Check in Accessibility api that all I can help for more information read https://stackoverflow.com/questions/35842762/how-to-read-window-content-using-accessibilityservice-and-evoking-ui-using-dra — ingsaurabh, Jul 24 '17 at 09:49
If you have it working now, could you post your answer then accept that answer? It will help those that find this question in the future. (And probably will help get some of those rep points back as well). — Gary99, Jul 25 '17 at 02:00
@Gary99 I can't post my code just yet, I am still having a few issues with it, for example it doesn't always click the text when an event happens. Or if multiple events happen, it will click the first text and then not the second text. — Jayce, Jul 26 '17 at 02:10

Abraham Toledo · Accepted Answer · 2017-07-28T09:47:37.533

I have done this using AccessibilityService. It will only work fine on API level >= 16 though.

You need to extend AccessibilityService. For instance this class will get text of USSD responses and dismiss the dialog.

// ....
public class UssdAccessibilityService extends AccessibilityService {
    public UssdAccessibilityService() {
    }

    @TargetApi(16)
    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
        if (!"com.android.phone".equalsIgnoreCase((String)event.getPackageName())){
            // In this example we are only interested in events comming
            // from "com.android.phone" package
            event.recycle();
            return;
        }

        String className = (String)event.getClassName();
        if (className == null || (!className.contains("AlertDialog") && !className.contains("AlertDialog"))){
            // Class is not an USSD dialog
            event.recycle();
            return;
        }

        AccessibilityNodeInfo source = event.getSource();
        if (source == null) {
            // getSource() is annotated @Nullable, so we do this to be
            // safe just in case
            event.recycle();
            return;
        }

        AccessibilityNodeInfo acceptButton = null;
        String ussdText = null;

        int childCount = source.getChildCount();
        for (int i = 0; i < childCount; i++){
            AccessibilityNodeInfo current = source.getChild(i);
            if (current == null)
                continue;

            String currentText = (String)current.getText();
            if (current.isClickable()){
                // In the case of USSD dialogs, there is only one clickable.
                // May be necessary to do more robust search in other scenarios
                acceptButton = current;
                continue;
            }

            ussdText = currentText;

            current.recycle();
        }

        if (ussdText!= null) {
            if (acceptButton != null)
                acceptButton.performAction(AccessibilityNodeInfo.ACTION_CLICK);

        }

        source.recycle();
        event.recycle();
    }

    // ....
}

You must declare the accessibility service in the manifest under <application>

<service
        android:name=".UssdAccessibilityService"
        android:enabled="true"
        android:label="Read USSD codes and dismiss"
        android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
        <intent-filter>
            <action android:name="android.accessibilityservice.AccessibilityService" />
        </intent-filter>

        <meta-data
            android:name="android.accessibilityservice"
            android:resource="@xml/accessibility_service_config" />
</service>

Under res/xml create accessibility_service_config.xml

<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:packageNames="com.android.phone,com.ats.android.activationcodebot"
    android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
    android:accessibilityFlags="flagDefault"
    android:accessibilityFeedbackType="feedbackSpoken"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
    />

Of course you have to adapt this code to your own needs.

Finally you will have to enable the accessibility service manually on Settings > Accessibility in Android (or ask the user to do it).

Read more ... Developing an Accessibility Service

I will try this as soon as I get home today and let you know if it works out for me. It is similar to the code I already have, but my code doesn't always "click" the screen for some reason. — Jayce, Jul 26 '17 at 15:38
I haven't seen your code but be sure that you iterate through the entire tree looking for the clickable view you want. — Abraham Toledo, Jul 26 '17 at 18:42
Also, be sure not to be using any instance or class field for temporary storage inside `onAccessibilityEvent()` as that would make it thread unsafe. If you really need to work with an instance field, please use `synchronized (lockObject){}` to make sure all calls to `onAccessibilityEvent()` execute secuentially. — Abraham Toledo, Jul 26 '17 at 18:49
Can you look at my updated code and maybe help me there? I profiled it off your code.. https://stackoverflow.com/questions/45398392/onaccessibilityevent-action-click-not-working — Jayce, Jul 30 '17 at 22:51

Android Clickable On screen text

1 Answers1