How to switch/change testInstrumentationRunner dynamically with gradle

Question

My project has 2 different groups of tests. One group runs only with the default AndroidJUnitRunner the other has to be run with a custom implementation TestRunner extends MonitoringInstrumentation.

Currently I switch the testInstrumentationRunner by editing the build.gradle each time I need to run the other group of tests:

android{
      defaultConfig {
          //testInstrumentationRunner "my.custom.TestRunner"
           testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
      }
}

I know that flavours can have their own testInstrumentationRunner but my current app already has 2 flavourDimensions. Using flavours is actually intended to have different versions of an app. I need 2 versions of the test application, both testing the same app with different testInstrumentationRunners.

I tried to change the testInstrumentationRunner by iterating over all test variants. There are actually multiple testInstrumentationRunner properties:

android.testVariants.all { TestVariant variant ->
    //readonly
    variant.variantData.variantConfiguration.instrumentationRunner

    variant.variantData.variantConfiguration.defaultConfig.testInstrumentationRunner

}

But as soon as android.testVariants is called the build gets configured and all changes are not reflected in the build.

How can I change the testInstrumentationRunner (from a gradle plugin) dynamically?

I'd prefer to have 2 different gradle tasks, each using a different testInstrumentationRunner but testing the same variant. Because I intent to create a gradle plugin the solution should work as plugin too.

Answer 1

Have you considered using console parameter as a switch between two configurations? As simple as that:

android {
      defaultConfig {
           if (project.ext.has("customRunner")) {
               testInstrumentationRunner "my.custom.TestRunner"
           } else {
               testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
           }
      }
}

And then for example run gradlew aDeb -PcustomRunner if you want to test using custom runner or gradlew aDeb to use default.

I know it's not rocket science but simpler is better, right? You can use it in your plugin too, just obtain the Project object and do the similar thing.

Answer 2

Since the android gradle plugin 1.3 it is possible to create separate test modules. Each of those test modules can have its own testInstrumentationRunner.

For a detailed example see the AndroidTestingBlueprint example project on github.

The solution from @johan-stuyts that got bounty works fine (or at least it did with the android gradle plugin 1.2). But it uses private APIs and creating a separate module is easier and future proof.

Answer 3

I had a similar issue, I used gradle's taskGraph. Based on your statement "My project has 2 different groups of tests." I'm going to assume you have different tasks defined, I'll call them testGroupOne and testGroupTwo:

task testGroupOne{
}
task testGroupTwo{
}
gradle.taskGraph.whenReady {taskGraph ->
    if(taskGraph.hasTask(testGroupOne)){
        testInstrumentationRunner "my.custom.TestRunner"
    } else if (taskGraph.hasTask(testGroupTwo)){
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
}

This enables you to set the value of testInstrumentationRunner after configuration but before execution.

Answer 4

This is a partial solution, but maybe this will make things a bit less cumbersome for you: It does not allow you to run the tests with both test runners in the same build. If you want that you would have to clone all task instances of DeviceProviderInstrumentTestTask, which, in my opinion, is too complex and fragile.

This works for me (Gradle 2.4 and Android build tools 1.2.3). It uses internal APIs, so it is possible that this no longer works with the next release of the Android build tools.

You should modify the tasks generated by the Android plug-in after the project has been evaluated. The changes will then be used by the test tasks. The property that actually changes the used test runner is task.testVariantData.variantConfiguration.testedConfig.mergedFlavor.testInstrumentationRunner. Instead of making these changes during the configuration phase (i.e. outside a task), the changes are applied using a task, so your test runner is only used when requested. By forcing the test tasks to run after useMyTestRunner (if the latter is part of the build), the class name of the test runner will have been changed when a test task starts:

project.afterEvaluate {
    task useMyTestRunner << {
        tasks.withType(com.android.build.gradle.internal.tasks.DeviceProviderInstrumentTestTask.class) { task ->
            task.testVariantData.variantConfiguration.testedConfig.mergedFlavor.testInstrumentationRunner = 'com.mycompany.MyTestRunner'
        }
    }

    tasks.withType(com.android.build.gradle.internal.tasks.DeviceProviderInstrumentTestTask.class) { task ->
        task.mustRunAfter useMyTestRunner
    }
}

You can now run the tests using the default test runner configured for the flavor(s) with:

gradle :myApp:connectedAndroidTest

When you want to run the tests with your test runner use:

gradle :myApp:connectedAndroidTest :myApp:useMyTestRunner

I did not add checks for null for any of the properties retrieved using task.testVariantData.variantConfiguration.testedConfig.mergedFlavor.testInstrumentationRunner. You should add them for robustness. I think at least testedConfig needs attention. See getInstrumentationRunner() at https://android.googlesource.com/platform/tools/build/+/master/builder/src/main/java/com/android/builder/VariantConfiguration.java

You can use an import for com.android.build.gradle.internal.tasks.DeviceProviderInstrumentTestTask at the top of your build file, so you only have to use the simple name of the class.