1

To check that I don't have transient errors in my tests I want to run my tests in random order (or at least change the order by myself). Currently I'm using IntelliJ IDEa for developing but I can't find an option to change the order of the tests.

Is there a way to achieve this?

Please note that I don't want to set the tests to run in a determined order, I want to try different orders or a randomized order independently of the tests order in the code. Besides, I can't modify the tests code, so the solution should be in a higher level (that is why I put as a tag IntelliJ IDEa). If it can't be done in IntelliJ, I am willing to use an external tool just to run the tests in a random order.

Martin Aparicio Pons
  • 183
  • 11
  • Please note: actually this is a **bad** idea. Assume your tests really fail on one certain order of your test cases. As a result of that, your nightly regression test suite fails like ... once per month. Probably without giving you much hint; so you have to spend a lot of time to figure out "in which order was stuff run"; to enable you to repro your problem (which will be really hard, because then, all of a sudden you have to run your test in a very fixed order). So, if your code is such that "random" matters; maybe you should be looking into other kinds of tests ... – GhostCat Jul 25 '16 at 17:04
  • For example, "rule based" checking: http://www.ontestautomation.com/an-introduction-to-property-based-testing-with-junit-quickcheck/ ... In other words: if you think that you benefit from random order; I strongly think you should be looking into "more than ordinary" unit tests. And if that is not the case, random order won't really help you. And worse, image unit tests owned by your project team. Everything works, and in two years from now, when you are long gone, that one sequence that causes trouble happens. Nobody changed anything, but stuff breaks. Just once. Poor soul fixing that. – GhostCat Jul 25 '16 at 17:10
  • 1
    I appreciate your comments, but the thing is that I'm working with an existing codebase and the tests normally have transient errors. We have automated test running but I can't test this issue locally. That is why I need to run the tests in random order locally, without changing code. – Martin Aparicio Pons Jul 25 '16 at 18:44

0 Answers0