Different test count depending on whether the tests pass successfully

Question

My project: testng-surefire-maven. In one of the modules I runt mvn clean install. When all tests are green, I have a result: Tests run: 277, Failures: 0, Errors: 0, Skipped: 0

Then I in turn make an intentional mistake in one of 3 tests that I am refactoring right now. And as a result I have 3 totally different outputs: Test1>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation Tests run: 344, Failures: 1, Errors: 0, Skipped: 100

Test2>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation Tests run: 282, Failures: 1, Errors: 0, Skipped: 8 Test3>AbstractTestNGSpringContextTests.springTestContextPrepareTestInstance:149 » BeanCreation Tests run: 416, Failures: 1, Errors: 0, Skipped: 205

How is that possible??? All I've done is a one-line change in one of the test classes in turn. I didn't touch testng.xml nor pom.xml.

Additionally, if I make a mistake in all 3 of them simultaneously, only one will pop up. I didn't set a custom skipAfterFailureCount in surefire nor any other testng property. Why doesn't it run though all of them and show me the list of all failing tests at once? All tests are in the same package.

Answer 1

Ok, I don't know what influences test count, but I can answer a second part of my question.

Additionally, if I make a mistake in all 3 of them simultaneously, only one will pop up. I didn't set a custom skipAfterFailureCount in surefire nor any other testng property. Why doesn't it run though all of them and show me the list of all failing tests at once? All tests are in the same package.

It happens because testng has a property - configfailurepolicy: Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. Default behavior is skip. And that's what happened in my case. I had problems on test init stage, not in the test method itself.