Run an instrument test from within app and wait for result

Question

I am developing a QA automation solution that can record/playback QA tests on Android. A key business requirement is to have no dependency on a connected PC while playing back the recorded test. To that end, I'm trying to run an Instrumentation test without a connected PC. (Specifically, an Appium UiAutomator2 test).

My current approach is trying to run the test programmatically from my app. If I were running the test normally from a connected PC, I would use the command adb shell am instrument -w. I tried accessing ADB Shell from my app and running am instrument -w, but this produces an error that I am missing the INTERACT_ACROSS_USERS_FULL permission.

To get around this issue, I am trying to run the test using startInstrumentation. This successfully starts the test. However, the test immediately crashes. Upon further investigation, I traced the crash to an NPE: the test is trying to retrieve InstrumentationRegistry.getInstrumentation.getUiAutomation(0), but this returns null.

How do I run the test programatically and give it access to the required UiAutomation instance?

This is how I start the test:

public void runTest() {
    final String pm = getPackageName().replaceFirst(".test$", "");
    final InstrumentationInfo info = getInstrumentationInfo(pm);
    if (info != null) {

        final ComponentName cn = new ComponentName(info.packageName,
                info.name);

        Bundle arguments = new Bundle();
        arguments.putString("class", "io.testim.appiumwrapper.test.AppiumUiAutomator2Server");
        //cn = {io.extension.test/android.support.test.runner.AndroidJUnitRunner}
        startInstrumentation(cn, null, arguments);

    } 
}

Do you want to wait for a pass or fail result for each test? — Jitesh Mohite, Aug 23 '18 at 03:33
I try to imitate Appium and run a test server which runs until I kill it. — Maor Hadad, Aug 23 '18 at 07:06
why you are not using "adb shell am instrument -w" within the app? — Jitesh Mohite, Aug 23 '18 at 15:48
You can't start adb shell command from app. Google block it and it won't work anymore. — Maor Hadad, Aug 23 '18 at 15:54
If I get over this, I will have the ability to run appium tests on every device that contain my testing app without the need to install appium on a pc. — Maor Hadad, Aug 23 '18 at 15:57
is it possible to run instrumentation test cases using appium? I never heared this. — Jitesh Mohite, Aug 23 '18 at 16:54
Have you seen https://stackoverflow.com/questions/14339216/how-to-start-instrumentation-project-programmatically-using-android-intent — Jitesh Mohite, Aug 23 '18 at 17:03
@jiteshmohite Yes. This is how I start it. The problem is I don't have the -w flag. — Maor Hadad, Aug 24 '18 at 09:17
@MaorHadad rather than using `adb shell` command from the app you could run the `am` command alone since the OS has it within the path. Ex: `am instrument -w` — ahasbini, Aug 27 '18 at 08:13
@ahasbini My first option was to run the am command via Runtime.getRuntime().exec(cmd). But I get this warning "Permission Denial: startInstrumentation asks to run as user -2 but is calling from user 0; this requires android.permission.INTERACT_ACROSS_USERS_FULL". This is the command I tried "am instrument -w io.extension.test/android.support.test.runner.AndroidJUnitRunner" — Maor Hadad, Aug 27 '18 at 16:27
how are you starting the instrumented test programmatically? — Sushant Somani, Aug 28 '18 at 05:20
I read your question three times, still didn't understand what your app exactly do, can you explain more about what the app does and what you are trying to test. maybe share few of your test cases as well — Alireza Ahmadi, Sep 19 '18 at 21:05
Also take a look at system primission to see if it helps you, it it does you can install your app as an system app (only available on rooted device) and get some extra permission. — Alireza Ahmadi, Sep 19 '18 at 21:06
You said you don't want to connect your phone to pc, how about Firebase Test Lab? It's a cloud platform by google that allows you to run instrumental test and it gives you 15 tests free per day but you can always buy more. — Alireza Ahmadi, Sep 19 '18 at 21:11

Martin Zeitler · Answer 1 · 2018-09-04T08:54:19.267

1

see signature protection level - clarifying ...unless having the package white-listed by Google's release key, you won't be able to obtain the necessary permission. this is a security/integrity feature, with the purpose to limit what malware is able to do - and what you intend to do there, is typical malware behavior - no matter the actual intention of it; working against the system leads nowhere.

the only way I could imaging would be to run commands directly from a terminal emulator or the test application - against a custom build of AOSP, so that you could add android:protectionLevel="signature" to the Manifest.xml and then require android.permission.INTERACT_ACROSS_USERS_FULL. but with a stock ROM, there definitely is no chance to do so. it is not, that it would be "impossible", but building a custom ROM means quite some effort, in order to get there. at least for Nexus and Pixel devices, the required drivers are available here; for other devices, you'd have to find them at the device's vendor, if they're even available.

the trick is to sign off the ROM with the same key as the app - only then signature level permissions can be obtained - while for a stock ROM, you'd (theoretically) need Google's release key to sign off the package. one can enforce single user, as explained here, while this also is only available to system apps.

edited Sep 04 '18 at 08:54

answered Sep 04 '18 at 08:10

Martin Zeitler

1
19
155
216

My use case (and bounty) for this is creating accessibility friendly apps that drive other apps. For example - I can record certain tough actions for a disabled person and then they'd play them. Although this can be considered problematic it's exactly the sort of thing the accessibility API should let me do (the user can opt in). The fact you _can_ start the test but you only can't wait for it indicates that this isn't _too_ impossible. In any case there is no problem with requiring the user to opt in but getting them to install another ROM wouldn't be practical. – Benjamin Gruenbaum Sep 04 '18 at 10:12
1

@BenjaminGruenbaum that bounty does not change anything to the technical requirements, which are defined by Android OS - which I've explained in detail. if this would be practical or applicable, was not part of the question. to me, the whole concept does not appear realistic - because it ignores the reality. using a x86 tablet PC, which runs an Android emulator might be the only alternate way to record interactions. – Martin Zeitler Sep 04 '18 at 10:19
1

I'll clarify. Starting UIAutomator2 does not add or subtract any capability from Android or OP's app. OP already uses AccessibilityService which (while requiring a permission) lets you technically do (almost) everything UIAutomator2 does. That is, it's possible to rewrite all this code on top of the APIs and permissions already used but that would be a huge waste of time. So this is certainly not "at odds" with what's possible. – Benjamin Gruenbaum Sep 04 '18 at 11:55
@BenjaminGruenbaum the error message should be something alike `java.lang.SecurityException: Permission Denial: get/set setting for user asks to run as user -2 but is calling from user 0`. this is not permitted for whatever package, while it is not able to obtain those `signature` level permissions, which only can be obtained by package signature. in order to become the OS, one needs to have the signing key of the OS. – Martin Zeitler Sep 04 '18 at 17:23
also since Android 5.0 the underlying system is SE Linux, while this here merely implements the idea of a "security context" (despite in Java), which cannot be provided; and it's about like trying to add a package into let's say the RedHat repository, which would also fail, even when able to upload it, while not having singed it off with the expected key. therefore the only way is to use pre-installed devices, where one can define which key would be expected, just alike an own distribution. – Martin Zeitler Sep 04 '18 at 17:36
Sadly I am familiar with signing both Android and my app as well as running as system (when being embedded as an SDK). It just sounds like a shame to implement all of the UIAutomator things on top of the accessibility API. – Benjamin Gruenbaum Sep 04 '18 at 21:14
@BenjaminGruenbaum using Gradle plugin `com.android.test` to create an instrumentation test module would be the only possible workaround here - because then both would run within the security context of the same user. https://developer.android.com/studio/test/ – Martin Zeitler Sep 04 '18 at 21:22
@MartinZeitler runInstrumintaionTest is a legit android API. It does run the test but produces a result like I ran it via ADB without the -w flag. I was wondering if there is a way to get the results as if I ran runInstrumintaionTest with the -w flag. – Maor Hadad Sep 05 '18 at 07:46

Run an instrument test from within app and wait for result

1 Answers1

Linked