1

I am running a suite of integration tests using maven and about 10% of the tests would fail or throw an error. However, when I start the server and run the individual failed tests manually from my IDE(intellij idea), they all pass with no problem. What could be the cause of this issue?

user3914823
  • 11
  • 1
  • 2
  • 3
    What is the error ? Maybe you're not cleaning your resources properly between each test. That could explain why they are passing when run individually but failing when run altogether. – m4rtin Aug 06 '14 at 14:44
  • I run with mvn clean install all the time so I think the resources are clean, or are you referring to cleaning the whole .m2 repo and run the tests again? – user3914823 Aug 06 '14 at 14:57
  • The failures and errors are really random. As I am testing a database, it is throwing errors like "table does not exist" or "SQL syntax error". – user3914823 Aug 06 '14 at 14:59
  • Hum no I was referring to the resources your code use, like database for example. The clean maven command only cleans generated files but I don't think that will help in your case (since everything compiles fine). Also do not clean your whole .m2 repo, I can't think of a situation where this will help. In any case, dependency management operations should be done via maven commands and/or pom configuration. Try to isolate an error (the "table does not exist" one for example), then try to find out why the table wasn't existing (probably because it is created somewhere else). – m4rtin Aug 06 '14 at 17:25
  • I cleaned up the whole database every time before I ran the test suite. This is really weird because table creation and deletion are all done by the integration tests and when run by maven they fail. Thank you for your help! I will try to dig into it more. – user3914823 Aug 06 '14 at 18:40

1 Answers1

2

This is almost always caused by the unit tests running in an inconsistent order, or a race condition between two tests running in parallel via forked tests. If Test #1 finishes first, it passes. But if Test #2 finishes first, it leaves a test resource, such as a test database, in an alternate state causing Test #1 to fail. It is very common with database tests, esepecially when one or more alter the database. Even in IDEA, you may find all the tests in the com.example.FooTest class always pass when you run that class. But if you run all the tests in the com.example package or all tests in the project, sometimes (or even always) a test in FooTest fails.

The fix is to ensure your tests are always guaranteed a consistent state when run. (That is a guiding principle for good unit tests.) You need to pay attention to test setup and tear-down via the @Before, @BeforeClass, @After, and @AfterClass annotations (or TestNG equivalents). I recommend Googling database unit testing best practices. For database tests, running tests in a Transaction can prevent these type of issues. That way the database is rolled back to its starting state whether the test passes or fails. Spring has some great support for JDBC dtaabase tests. (Even if your project is not a Spring project, the classes can be very useful.) Read section 11.2.2 Unit Testing support Classes and take a look at the AbstractTransactionalJUnit4SpringContextTests / AbstractTransactionalTestNGSpringContextTests classes and the @TransactionConfiguration annotation (this latter one being from running Spring Contexts). There are also other Database testing tools out there such as DbUnit.

Javaru
  • 30,412
  • 11
  • 93
  • 70
  • Any way to force the tests to be run always in the same order ? – Stephane Jan 05 '16 at 13:05
  • IDEA does not as it has a philosophy of following best practices and not allowing "less than ideal practices". Ideally unit tests should be independent of each other. The better solution would be to use a @BeforeTest/setup() and/or @AfterTest/teardown() method to get the test environment into a know baseline state. And if IDEA allowed tests to be run in a particular order, different IDEs or build systems would still have issues. That said, JUnit 4.11 added the ability have some control over the order tests are run via @FixMethodOrder. See answer #2 at http://stackoverflow.com/q/3693626 – Javaru Jan 05 '16 at 14:25
  • Yes, I agree with you. It's just that I have a series of tests that fails. I'm looking for ideas on fixing it. – Stephane Jan 05 '16 at 16:49