AndroidJUnitRunner unit tests are very slow to execute

Question

I'm using the AndroidJUnitRunner in my project, but the unit tests are painfully slow to execute on a device and emulator both in Android Studio and from the command-line via gradlew.

I'm using the master branch of this open-source project, OneBusAway (as of this commit): https://github.com/OneBusAway/onebusaway-android

I'm seeing execution time of all 142 tests that is upwards of 3 minutes in real elapsed time, but Android only registers a smaller portion of this in the execution time it shows under "Test Results". Before switching to the AndroidJUnitRunner all of these unit tests executed under 20 seconds consistently.

Here's what an example test looks like:

/**
 * Tests to evaluate utility methods related to math conversions
 */
@RunWith(AndroidJUnit4.class)
public class MathUtilTest {

    @Test
    public void testOrientationToDirection() {
        // East
        double direction = MathUtils.toDirection(0);
        assertEquals(90.0, direction);
    }
}

Here's the build.gradle config:

android {
    dexOptions {
        preDexLibraries true
    }
    compileSdkVersion this.ext.compileSdkVersion
    buildToolsVersion "27.0.3"

    defaultConfig {
        minSdkVersion 14
        targetSdkVersion 21
        versionCode 93
        versionName "2.3.8"

        multiDexEnabled true

        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"

        // The following argument makes the Android Test Orchestrator run its
        // "pm clear" command after each test invocation. This command ensures
        // that the app's state is completely cleared between tests.
        testInstrumentationRunnerArguments clearPackageData: 'true'
    }
    ...
    testOptions {
        execution 'ANDROID_TEST_ORCHESTRATOR'
        unitTests.includeAndroidResources true
    }
...
}
dependencies {
...
    // Unit tests
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestUtil 'com.android.support.test:orchestrator:1.0.2'
...
}

Why is this running unit tests so slow?

score 5 · Accepted Answer · answered Jul 06 '18 at 19:03

Apparently the slowdown is related to including a dependency on Android Test Orchestrator, which I don't need (even if I'm running tests on a device that require an Android context). It wasn't clear to me from the existing documentation that Orchestrator wasn't required for these tests.

I removed the following lines from build.gradle, and my total execution time via Android Studio dropped back down into the ~15 second range:

// The following argument makes the Android Test Orchestrator run its
// "pm clear" command after each test invocation. This command ensures
// that the app's state is completely cleared between tests.
testInstrumentationRunnerArguments clearPackageData: 'true'
...
execution 'ANDROID_TEST_ORCHESTRATOR'
...
androidTestUtil 'com.android.support.test:orchestrator:1.0.2'

So here's the new build.gradle that executes tests in ~15 seconds:

android {
    dexOptions {
        preDexLibraries true
    }
    compileSdkVersion this.ext.compileSdkVersion
    buildToolsVersion "27.0.3"

    defaultConfig {
        minSdkVersion 14
        targetSdkVersion 21
        versionCode 93
        versionName "2.3.8"

        multiDexEnabled true

        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    ...
    testOptions {
        unitTests.includeAndroidResources true
    }
    ...
}
...
dependencies {
    ...
    // Unit tests
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
}

I thought that maybe the testInstrumentationRunnerArguments clearPackageData: 'true' was the primary culprit causing increased execution time to clear the package data, but removing that line alone didn't change the total execution time of tests - I had to completely remove the Orchestrator dependency.

Here's the commit on Github that removed Orchestrator: https://github.com/OneBusAway/onebusaway-android/commit/a1657c443258ac49b1be83a75399cf2ced71080e

Our experience with the orchestrator was that we required it for our test suite to clear down the app. But unfortunately now the speed is far slower as you experienced. Hopefully the Android dev team will be able to improve on this in future releases. — Eurig Jones, Aug 03 '18 at 09:07

AndroidJUnitRunner unit tests are very slow to execute

1 Answers1