0

My tests are executed in different order in eclipse and in jenkins. I want the results to be the same and independent from the environment. The tests are not parallel. I do not care about specyfing the order, I just want them to run in the same order always.

Why? Because they are integration tests and they use the database. Sometimes they impact each other and it would be hard to isolate them completely. So if they fail, I'd like at least to have them failing everywhere to make fixes easier, without debugging jenkins remotely...

edit: here is an example of the order i'm getting in eclipse:

2013-01-22 14:39:06,186 main INFO  category  - Starting test A.a
2013-01-22 14:39:06,547 main INFO  category  - Starting test B.a
2013-01-22 14:39:10,614 main INFO  category  - Starting test C.a
2013-01-22 14:39:11,983 main INFO  category  - Starting test D.a
2013-01-22 14:39:12,492 main INFO  category  - Starting test D.b
2013-01-22 14:39:12,889 main INFO  category  - Starting test A.b
2013-01-22 14:39:13,657 main INFO  category  - Starting test A.c
2013-01-22 14:39:18,626 main INFO  category  - Starting test D.c
2013-01-22 14:39:19,041 main INFO  category  - Starting test A.d
2013-01-22 14:39:19,756 main INFO  category  - Starting test A.e
2013-01-22 14:39:20,724 main INFO  category  - Starting test D.e
2013-01-22 14:39:21,515 main INFO  category  - Starting test A.f

I'm using groups in test suite xml (actually, in this particular suite I run all except some groups). There is no specific list of test methods provided in the xml. The order above is quite random.

mabn
  • 2,473
  • 1
  • 26
  • 47
  • 1
    In [TestNG documentation](http://testng.org/doc/documentation-main.html): *By default, TestNG will run your tests in the order they are found in the XML file. If you want the classes and methods listed in this file to be run in an unpredictible order, set the preserve-order attribute to false*. – assylias Jan 22 '13 at 12:53
  • possible duplicate of [order of execution of tests in testng](http://stackoverflow.com/questions/2669576/order-of-execution-of-tests-in-testng) – Don Roby Jan 22 '13 at 12:54
  • @DonRoby this link is unrelated, I explicitly mentioned that any order suits me if it will be consisdent across environments. – mabn Jan 22 '13 at 13:48
  • If you can specify the order, it will be consistent. The link is thus certainly not unrelated. I accept that it may not be an *exact* duplicate. – Don Roby Jan 22 '13 at 14:03
  • Well, maybe. But suggestions from that link do not work for me. Priority does not work across classes (I think it orders only methods in a given class). dependsOnMethods has the same issue - if b() depends on a() the execution often looks like this: a(), OtherClass.x(), b(). – mabn Jan 22 '13 at 14:41

2 Answers2

0

Ideally you should not rely on ordering of unit tests. Each test should be independent from each other. Going by this philosophy, JUnit does not guarantee any specific execution order of Unit Tests. You can take a look at TestNG which supports executing the tests in a specific order.

MoveFast
  • 3,011
  • 2
  • 27
  • 53
  • Yea, but these are not unit tests ("Why? Because they are integration tests and they use the database") – mabn Jan 22 '13 at 14:35
0

If you want your classes / methods to be run in a predictable order, then you should use the preserve-order attribute in TestNG.

 <suite name="Preserve order test runs">
   <test name="Test 1" preserve-order="true">
    <classes>
     <class name="com.ClassOne"/>
     <class name="com.ClassTwo"/>
     <class name="com.ClassThree"/>
    </classes>
   </test>
 </suite>
AnthonyW
  • 1,910
  • 5
  • 25
  • 46
Vinod Baradwaj
  • 118
  • 8