Can AccessibilityService dispatch key events, including even Unicode characters?

Question

Background

I want to try to send Unicode characters from the PC to the Android device via adb commands, as if they are being typed from a physical keyboard. Characters from various languages, for example, and not just English.

The problem

Such a thing is impossible using the commands I've found, as it seems to supports only a basic set of characters (probably only Ascii) :

adb shell "input keyboard text 'This goes to Android device'"

Because of this, I've decided to request it to be supported, here (please consider starring).

As a workaround, I thought that maybe I could develop an app that uses AccessibilityService and it would dispatch key events as if I'm typing via the device, and the PC would send such events using adb directly to the app via an Intent.

Thing is, after creating the app, I can't find which function I should use to do it.

What I've found

There are multiple things I've found:

1. onAccessibilityEvent - this is not for dispatching. It's only for getting events, which I don't think I will even need in this case.
2. getSoftKeyboardController - a function that can help with hiding the automatically shown keyboard, but that's about it...
3. dispatchGesture - a function that seems to be used only for dispatching touch events. It seems quite cool, but I don't see that it can handle keys.
4. performGlobalAction - seems promising, but sadly supports a very limited set of operations (back-key, home-key, etc...).
5. findFocus - I think I could use this and then dispatch a key event on what I get, but I'm not sure if this is a valid way to do it as I want to dispatch the event globally (plus maybe I would get null object, which means it might not be reliable). Not to mention that according to the options I see, it doesn't allow me to put the text right on the caret and that's it.

The question

Is it possible for AccessibilityService to dispatch a key event of Unicode characters, as if I type some text?

What's the best option to use for this?

Answer 1

This is an unconventional solution.

You can use the UI Automator framework to send Unicode characters through ADB to a focused text field like the input command does with ASCII characters (but fails with Unicode characters.)

First, implement an Android automation test that is capable of receiving broadcasts. Broadcasts will direct the test to do certain tasks. The implementation below will clear text and enter text using Base-64 or Unicode. I should not that the following can act like a background server until stopped.

AdbReceiver.kt

package com.example.adbreceiver

/*
 * Test that runs with a broadcast receiver that accepts commands.
 *
 * To start the test:
 * adb shell nohup am instrument -w com.example.adbreceiver.test/androidx.test.runner.AndroidJUnitRunner
 *
 * On Windows, the code page may need to be changed to UTF-8 by using the following command:
 *      chcp 65001
 *
 */
import android.content.BroadcastReceiver
import android.content.Context
import android.content.Intent
import android.content.IntentFilter
import android.util.Base64
import androidx.test.core.app.ApplicationProvider
import androidx.test.ext.junit.runners.AndroidJUnit4
import androidx.test.filters.SdkSuppress
import androidx.test.platform.app.InstrumentationRegistry
import androidx.test.uiautomator.By
import androidx.test.uiautomator.UiDevice
import org.junit.Test
import org.junit.runner.RunWith

@RunWith(AndroidJUnit4::class)
@SdkSuppress(minSdkVersion = 18)
class AdbInterface {
    private var mDevice: UiDevice? = null
    private var mStop = false

    private val ACTION_MESSAGE = "ADB_INPUT_TEXT"
    private val ACTION_MESSAGE_B64 = "ADB_INPUT_B64"
    private val ACTION_CLEAR_TEXT = "ADB_CLEAR_TEXT"
    private val ACTION_STOP = "ADB_STOP"

    private var mReceiver: BroadcastReceiver? = null

    @Test
    fun adbListener() {
        mDevice = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
        if (mReceiver == null) {
            val filter = IntentFilter(ACTION_MESSAGE)
            filter.addAction(ACTION_MESSAGE_B64)
            filter.addAction(ACTION_CLEAR_TEXT)
            filter.addAction(ACTION_STOP)
            mReceiver = AdbReceiver()
            ApplicationProvider.getApplicationContext<Context>().registerReceiver(mReceiver, filter)
        }
        try {
            // Keep us running to receive commands.
            // Really not a good way to go, but it works for the proof of concept.
            while (!mStop) {
                Thread.sleep(10000)
            }
        } catch (e: InterruptedException) {
            e.printStackTrace()
        }
    }

    fun inputMsg(s: String?) {
        mDevice?.findObject(By.focused(true))?.setText(s)
    }

    internal inner class AdbReceiver : BroadcastReceiver() {
        override fun onReceive(context: Context, intent: Intent) {
            when (intent.action) {
                ACTION_MESSAGE -> {
                    val msg = intent.getStringExtra("msg")
                    inputMsg(msg)
                }
                ACTION_MESSAGE_B64 -> {
                    val data = intent.getStringExtra("msg")
                    val b64 = Base64.decode(data, Base64.DEFAULT)
                    val msg: String
                    try {
                        msg = String(b64, Charsets.UTF_8)
                        inputMsg(msg)
                    } catch (e: Exception) {
                    }
                }

                ACTION_CLEAR_TEXT -> inputMsg("")

                ACTION_STOP -> {
                    mStop = true
                    ApplicationProvider.getApplicationContext<Context>()
                        .unregisterReceiver(mReceiver)
                }
            }
        }
    }
}

Here is a short demo running on an emulator. In the demo, the first text "你好嗎? Hello?" is entered with adb using Base-64 encoding. The second text, "你好嗎? Hello, again?" is entered as a straight Unicode string.

Here are three Windows .bat file to manage the interface. There is not reason that these can't be ported to other OSes.

start.bat

Starts the instrumented test that receives the command broadcasts. This will run until it receives the "ADB_STOP" command.

rem Start AdbReceiver and disconnect.
adb shell nohup am instrument -w com.example.adbreceiver.test/androidx.test.runner.AndroidJUnitRunner

send.bat

Used for the demo to send Unicode text but can be easily generalized

rem Send text entry commands to AdbReceiver. All text is input on the current focused element.
rem Change code page to UTF-8.
chcp 65001 
rem Clear the field.
adb shell am broadcast -a ADB_CLEAR_TEXT
rem Input the Unicode characters encode in Base-64.
adb shell am broadcast -a ADB_INPUT_B64 --es msg 5L2g5aW95ZeOPyBIZWxsbz8=
rem Input the Unicode characters withouth further encoding.
adb shell am broadcast -a ADB_INPUT_TEXT --es msg '你好嗎? Hello, again?'

stop.bat

Stops the instrumented test.

rem Stop AdbReceiver.
adb shell am broadcast -a ADB_STOP

For some reason, the code only works on API 21+. It doesn't error out on earlier APIs but just silently fails.

This is just a proof of concept and the code needs more work.

Project AdbReceiver is on GitHub.

How to Run (Windows)

1. Start an emulator.

2. Bring up the project in Android Studio.

3. Under java->com.example.adbreceiver (andoidTest) right click AdbInterface.

4. In the pop-up menu, click "Run". This will start the instrumented test.

5. Bring up any app on the emulator and set the cursor into a data entry fields (EditText).

6. In a terminal window, enter

   adb shell am broadcast -a ADB_INPUT_TEXT --es msg 'Hello World!'

This should enter "Hello World!" into the text field.

This can also be accomplished from the command line. See the "start.bat" file above on how to start the test and the "stop.bat" file on how to stop it.

Notes on UI Automator and Assessibility

I took a look under the hood at how UI Automator works. As the OP guessed, UI Automator does use assessibility services on Android. The AssessibilityNode is used to set text. In the posted code above, the inputMesg() function has the line:

mDevice?.findObject(By.focused(true))?.setText(s)

findObject() is in UiDevice.java which looks like this:

public UiObject2 findObject(BySelector selector) {
    AccessibilityNodeInfo node = ByMatcher.findMatch(this, selector, getWindowRoots());
    return node != null ? new UiObject2(this, selector, node) : null;
}

setText() can be found in _UiObject2.java` and starts off like this:

public void setText(String text) {
        AccessibilityNodeInfo node = getAccessibilityNodeInfo();

        // Per framework convention, setText(null) means clearing it
        if (text == null) {
            text = "";
        }

        if (UiDevice.API_LEVEL_ACTUAL > Build.VERSION_CODES.KITKAT) {
            // do this for API Level above 19 (exclusive)
            Bundle args = new Bundle();
            args.putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, text);
            if (!node.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)) {
                // TODO: Decide if we should throw here
                Log.w(TAG, "AccessibilityNodeInfo#performAction(ACTION_SET_TEXT) failed");
            }
        } else {
        ...

So, accessibility services is integral to UI Automator.

Answer 2

Here is what I found based on my research on input command.

input keyevent KEYCODE_N

These KEYCODE are predefined in here

Logs:

Input   : injectKeyEvent: KeyEvent { action=ACTION_DOWN, keyCode=KEYCODE_N, scanCode=0, metaState=0, flags=0x0, repeatCount=0, eventTime=25861196, downTime=25861196, deviceId=-1, source=0x101 }
Input   : injectKeyEvent: KeyEvent { action=ACTION_UP, keyCode=KEYCODE_N, scanCode=0, metaState=0, flags=0x0, repeatCount=0, eventTime=25861196, downTime=25861196, deviceId=-1, source=0x101 }

To get more insight on above logs refer this source file.

Even switching the keyboard layout to other languages in the device, the keyevent only prints the ASCII characters.

Here are the resources that I found for key event in accessibility services:

In Android framework the accessibility service already dispatch the key Event. Refer here

FLAG_REQUEST_FILTER_KEY_EVENTS: If this flag is set the accessibility service will receive the key events before applications allowing it implement global shortcuts.

Note: Analysis based on Android 9.0

Answer 3

Not sure, if I exactly understand what you are looking for.

But, here are some resources that you can refer to perform actions on behalf of users using the Accessibility Service.

1. performAction()
2. performGlobalAction()
3. Taking Actions for User

Or in case you are looking to develop a completely new accessibility service, you can refer to this:

• Developing an Accessibility Service for Android