I am having a problem where unit tests are modifying the static state of class Foo and not cleaning up after themselves. This means that a unit test may pass when it is run by itself, but fail when some other unit test runs before it, because the state has changed.
I've been going through the code looking for those tests that change the static state of Foo and adding the necessary clean ups, but then I had another idea: why not use a junit test listener to examine the static state of Foo after each test has run, and fail if Foo is left in a modified state? Then I automatically get a list of naughty tests with each build, and can identify any similar problems in the future.
So I implemented a junit RunListener and modified my pom.xml to add the listener to the maven-surefire-plugin, and verified that the listener is executed for each test case.
The simple solution is to override the testFinished() method of RunListener and check the static state:
public class FooChecker extends RunListener {
@Override
public void testFinished(Description description) throws Exception {
if (fooClassInModifiedState()) {
throw new IllegalStateException("Test " + description.getDisplayName()
+ " has left Foo in a modified state");
}
}
}
This will work on tests of the following form:
public class SomeTest {
@Before
public void setUpTest() {
Foo.modifyState();
}
@After
public void tearDownTest() {
Foo.resetState();
}
@Test
public void testCase1() {
...
}
@Test
public void testCase2() {
...
}
}
After each test case, the @After method is invoked and cleans up the state. Then the testFinished() method of the junit run listener is invoked, and verifies that the state has been reset. If I remove the @After method, then the run listener will raise an error, as expected.
The problem here is that the testFinished method of the junit run listener is called after the @BeforeClass and before any @AfterClass methods of the unit test. If the unit test modifies the state of Foo in a @BeforeClass and cleans up in an @AfterClass method, then the testFinished method of the listener will see the modified state and incorrectly raise an error:
public class SomeTest {
@BeforeClass
public void setUpClass() {
Foo.modifyState();
}
@Test
public void testCase1() {
...
}
@Test
public void testCase2() {
...
}
@AfterClass
public void tearDownClass() {
Foo.resetState();
}
}
Here, the testFinished() method of the run listener is invoked after testCase1() and testCase2() have been invoked, but before the @AfterClass method is invoked, so the state of Foo is still modified, and the run listener will raise an error. That is not the correct behaviour.
In general, for two unit tests TestA and TestB, and a junit run listener Listener, the calling sequence seems to be:
- Listener.testRunStarted()
- TestA.@BeforeClass
- Listener.testStarted("TestA.testCase1)
- TestA.@Before
- TestA.testCase1()
- TestA.@After
- Listener.testFinished(TestA.testCase1)
- Listener.testStarted("TestA.testCase2)
- TestA.@Before
- TestA.testCase2()
- TestA.@After
- Listener.testFinished(TestA.testCase2)
- TestA.@AfterClass
- TestB.@BeforeClass
- Listener.testStarted("TestB.testCase1)
- TestB.@Before
- TestB.testCase1()
- TestB.@After
- Listener.testFinished(TestB.testCase1)
- TestB.@AfterClass
- Listener.testRunFinished()
What I'd like is to perform the check of modified state after the @AfterClass method has been invoked for a unit test, and before the @BeforeClass method is invoked on the next unit test. This way I can ensure that a unit test (if not the individual test cases) perform the necessary cleanup and still respect the test semantics.
But I can't see a way to do this using the junit RunListener and invoking tests using maven surefire plugin.
Is there a solution here?
edit: Adding a junit class rule to existing tests that checks the state is not an option, because I would need to add it to every test that exists now and every test that is added in the future. I want something that will automagically discover any violations, if possible.
edit: the underlying issue here is that each unit test for the maven module is run in the same JVM. I tried running each unit test in its own JVM (via setting the forking options of surefire) but the tests took too long to run. So I am stuck with a single JVM for multiple unit tests.
edit: Of course using static methods is bad design in the first place, and refactoring to inject state would be preferred, but refactoring a legacy code base of 1,000,000+ lines is not an option at this stage.
