How to continue an Android instrumentation test run after exception?

Question

Apparently an instrumentation test run is stopped when on exception occurs in the instrumented application:

Test failed to run to completion. Reason: 'Instrumentation run failed due to 'Exception''. Check device logcat for details

Is this the desired behavior or a misconfiguration in a custom instrumentation runner?

I'm using a custom MonitorinInstrumentation [1] in order to automate acceptance tests.

Unfortunately test execution is canceled when on exception occurs in one test. I want the test suite to complete and only mark the failed tests, like in JUnit.

Is there a general approach to execute (connected) tests without quitting the whole instrumentation in case an exception occurs in one test?

https://github.com/cucumber/cucumber-jvm/blob/master/examples/android/android-studio/Cukeulator/app/src/androidTest/java/cucumber/cukeulator/test/Instrumentation.java

score 2 · Answer 1 · answered May 01 '16 at 16:42

2

Instrumentation tests raise an Exception when something goes wrong (e.g. some conditions that you want to check). You can usually avoid some test to fail using try catch statement(or changing those checks). In this case there's something that made Dalvik Virtual Machine stop. This is usually caused by a crash in your app. Try to check carefully your tests flow to analyze if there are some crashes. Also, be sure to not use System.exit(0) in onDestroy() in some of your activities because this can cause your problem. I hope to can help you.

answered May 01 '16 at 16:42

Lorenzo Camaione

505
3
15

First of all many thanks for your answer. Unfortunately it does not provide a solution to my problem. As I've said, the application under tests raises the exception which stops the complete instrumentation run. Adding a try-catch in the test is not applicable. I'm looking for a solution - if it's possible with instrumentation at all - where instrumentation run continues and just the test, which causes the application to crash, is marked "red". – André Diermann May 02 '16 at 08:11
I know it's not a solution for this problem. As I've said there's something stoppig Dalvik Virtual Machine: in my opinion this can be related to some error in your code and so I invite you to check your code. Look for some crashes and be sure to not call **System.exit(0)** in **onDestroy()**. – Lorenzo Camaione May 02 '16 at 08:34
Neither System.exit() nor onDestroy() is called. But Dalvik VM stops, of course, due to the exception in the instrumented application. After reading http://developer.android.com/tools/testing/testing_android.html#Instrumentation I was under the impression that the instrumentation starts the application process and is somehow independent of the application. But you are saying, that the instrumentation "dies" if the instrumented process "dies"? Is there any chance to decouple this? – André Diermann May 02 '16 at 13:18
No there isn't. If you look at Gradle tasks you will notice that same tasks run when you normally run your app or when you run it under Instrumentation tests. The difference is that when you run your app under test some other tasks run. Definitely, I should look at your code to help you more. – Lorenzo Camaione May 02 '16 at 13:34
1

Unfortunately I could not share the actual code due to NDA, however this example has a similar setup https://github.com/a11n/android-cucumber-espresso/tree/stackoverflow ... just execute ./gradlew connectedCheck – André Diermann May 02 '16 at 13:59
I've just run your Instrumentationtest and I haven't encountered any problem. I have a green "ScenarioUserNotification" test. – Lorenzo Camaione May 02 '16 at 16:16
I created a branch "stackoverflow" for this (see link)... maybe you cloned the repository and builded the main branch accidentally? Please type "git branch" in Android Studio terminal... if you are on "main" branch type "git checkout stackoverflow". Sorry for inconvenience. Do you see the exception now? – André Diermann May 03 '16 at 12:11

score 1 · Answer 2 · answered May 04 '16 at 22:18

1

You can try to put in your root build.gradle

tasks.withType(Test) {
  ignoreFailures = true
}

According to the documentation:

If one of the test fails, on any device, the build will fail.

Check this android testing

answered May 04 '16 at 22:18

newhouse

946
8
12

1

My problem is not a failing build. Hence, your proposal unfortunately won't help. (My problem is, that the test suite does not run to completion if one test leads to an exception within the application under test.) – André Diermann May 10 '16 at 07:02

score 0 · Answer 3 · answered Feb 06 '20 at 23:49

I was recently looking for a solution to this problem and found some info that other might find useful.

Enter org.junit.rules.ErrorCollector. While this won't work if your instrumentation tests are using deprecated ApplicationTestCase<Application> class, I was able to switch to using androidx.test.platform.app.InstrumentationRegistry to get around this issue since I only needed the application context.

Here is some example code from a gist I found to help get started with:

package com.foo;

import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ErrorCollector;

import static org.hamcrest.Matchers.equalTo;

/**
 * Created: 12/04/19 17:12
 *
 * @author chris
 */
public class MyTest {
    @Rule
    public ErrorCollector collector = new ErrorCollector();

    @Test
    public void myTest() {
        collector.checkThat("a", equalTo("b"));
        collector.checkThat(1, equalTo(2));
    }
}

score 0 · Answer 4 · answered Feb 27 '23 at 03:25

You can try Android Test Orchestrator (I didn't try it.).

According to the documentation:

Crashes are isolated: Even if one test crashes, it takes down only its own instance of Instrumentation. This means that the other tests in your suite still run, providing complete test results.

See Android Test Orchestrator

How to continue an Android instrumentation test run after exception?

4 Answers4