changing order of unit tests in Python

Question

How can I make it so unit tests in Python (using unittest) are run in the order in which they are specified in the file?

It also precludes running the tests in parallel on multicores, or distributed across a cluster. Not a good idea. — Jörg W Mittag, Oct 24 '10 at 03:50
It would matter if 1. You need to debug and want to run the tests in order of increasing complexity and 2. If a small percentage of the tests are computationally heavy, but there are errors that can be spotted with the simpler tests, then having them ordered will make your work much, much faster. — Michael Clerx, Oct 22 '12 at 11:31
What is the Python standard for integration testing then? Is it not...unittest? — andy, Apr 21 '14 at 18:10

score 29 · Accepted Answer · answered Oct 23 '10 at 21:02

29

You can change the default sorting behavior by setting a custom comparison function. In unittest.py you can find the class variable unittest.TestLoader.sortTestMethodsUsing which is set to the builtin function cmp by default.

For example you can revert the execution order of your tests with doing this:

import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)

answered Oct 23 '10 at 21:02

atomocopter

948
10
20

14

This does not answer the question above – Udi Aug 28 '13 at 21:48
3

This *does* answer the question above – chown Mar 07 '15 at 22:37
1

This does seem to do the trick. So is the default order always to run opposite the order of declaration? – BenB Jul 31 '15 at 22:09
The documentation indicates it depends on the default sorting of strings:--- Note that the order in which the various test cases will be run is determined by sorting the test function names with respect to the built-in ordering for strings. – Moshe Zvi Jun 27 '18 at 00:43

score 23 · Answer 2 · answered Oct 23 '10 at 19:36

23

Clever Naming.

class Test01_Run_Me_First( unittest.TestCase ):
    def test010_do_this( self ):
        assertTrue( True )
    def test020_do_that( self ):
        etc.

Is one way to force a specific order.

answered Oct 23 '10 at 19:36

S.Lott

384,516
81
508
779

4

Slightly more clever name is Test010_Run_Me_First so if you decided to push a new one between two, you could. – Mansour Feb 06 '12 at 16:59
1

Indeed, clever :). – ivanleoncz Nov 27 '18 at 21:46

score 8 · Answer 3 · answered Aug 28 '13 at 22:10

As said above, normally tests in test cases should be tested in any (i.e. random) order.

However, if you do want to order the tests in the test case, apparently it is not trivial. Tests (method names) are retrieved from test cases using dir(MyTest), which returns a sorted list of members. You can use a clever (?) hack to order methods by their line numbers. This will work for one test case:

if __name__ == "__main__":
    loader = unittest.TestLoader()
    ln = lambda f: getattr(MyTestCase, f).im_func.func_code.co_firstlineno
    lncmp = lambda a, b: cmp(ln(a), ln(b))
    loader.sortTestMethodsUsing = lncmp
    unittest.main(testLoader=loader, verbosity=2)

Nice trick, but there's an error in your code. It should be ``lncomp = lambda _, a, b: cmp(ln(a), ln(b))`` — saroele, Nov 14 '13 at 11:38

score 2 · Answer 4 · edited May 23 '17 at 12:09

2

Use proboscis library as I mentioned already (please see short description there).

edited May 23 '17 at 12:09

Community

1
1

answered Feb 07 '13 at 14:05

Sergei Danielian

4,938
4
36
58

Tried proboscis... it's worth mentioning that it hijacks the test loading process so if you're using nose, any nose plugins that take advantage of that (e.g. testid) will stop working – Clintm Apr 01 '15 at 22:01

score 2 · Answer 5 · answered Oct 23 '10 at 20:31

2

There are also test runners which do that by themselves – I think py.test does it.

answered Oct 23 '10 at 20:31

Felix Schwarz

2,938
4
28
41

score 1 · Answer 6 · answered Apr 29 '20 at 09:39

1

The default order is alphabetical. You can put test_<int> before every test to specify the order of execution.

Also you can set unittest.TestLoader.sortTestMethodsUsing = None to eliminate the sort.

Check out Unittest Documentation for more details.

answered Apr 29 '20 at 09:39

Ognian Baruh

53
8

score 0 · Answer 7 · answered May 07 '15 at 17:57

I found a solution for it using PyTest ordering plugin provided here.

Try py.test YourModuleName.py -vv in CLI and the test will run in the order they have appeared in your module.

I did the same thing and works fine for me.

Note: You need to install PyTest package and import it.

score 0 · Answer 8 · answered Jul 06 '18 at 18:08

hacky way (run this file in pycharm or other unit test runner)

#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import unittest


def make_suite():
    class Test(unittest.TestCase):
        def test_32(self):
            print "32"
        def test_23(self):
            print "23"

    suite = unittest.TestSuite()
    suite.addTest(Test('test_32'))
    suite.addTest(Test('test_23'))
    return suite

suite = make_suite()

class T(unittest.TestCase):
    counter = 0
    def __call__(self, *args, **kwargs):
        res = suite._tests[T.counter](*args, **kwargs)
        T.counter += 1
        return res

for t in suite._tests:
    name = "{}${}".format(t._testMethodName, t.__class__.__name__)
    setattr(T, name, t)

changing order of unit tests in Python

8 Answers8

Linked