83

I want to run all my pytest tests in parallel instead of sequentially.

my current setup looks like:

class Test1(OtherClass):
    @pytest.mark.parametrize("activity_name", ["activity1", "activity2"])
    @pytest.mark.flaky(reruns=1)
    def test_1(self, activity_name, generate_test_id):
    """
    """

        test_id = generate_random_test_id()
        test_name = sys._getframe().f_code.co_name

        result_triggers = self.proxy(test_name, generate_test_id, test_id, activity_name)

        expected_items = ["response"]
        validate_response("triggers", result_triggers, expected_items)


    @pytest.mark.parametrize("activity_name", ["activity1", "activity2"])
    @pytest.mark.flaky(reruns=1)
    def test_2(self, activity_name, generate_test_id):
    """
    """

        #same idea...

I run my tests using pytest -v -s.

The result is that my tests are running sequentially, which takes a lot of time since some of them wait for responses from remote servers (integration tests).

Is there any way of running pytest in parallel?

Marcus Müller
  • 34,677
  • 4
  • 53
  • 94
Noy Mizrahi
  • 831
  • 1
  • 6
  • 4

3 Answers3

81

You want pytest-xdist. I think Qxf2 explains it quite well: Qxf2 on Pytest-Xdist

Their Linux command-line is slightly too verbose for my tastes though; I use:

pytest -n <NUM>

where <NUM> is the number of parallel workers.

Claire Nielsen
  • 1,881
  • 1
  • 14
  • 31
  • 2
    Hi @Claire Nielsen. From your experience can pytest be executed for an arbitrary `n` (whatever that means for the final performance) or is it constrained only to 4 or 8 or how many CPU cores you have in a given system ?.. – George Pligoropoulos Aug 11 '22 at 06:30
42

UPDATE: The solution below is to demonstrate concurrent pytests are possible, but pytest-parallel is currently unmaintained.

pytest-xdist is a great solution for most cases, but integration tests are special. After sending a request to a remote server, another test can start on a new thread instead of waiting for a response. This is concurrent testing instead of parallel. Concurrency allows many more tests at once with much less memory and processing overhead.

I wrote the pytest-parallel plugin [py3.6+] to enable parallel and concurrent testing. Here's how to run your integration tests concurrently:

pytest --tests-per-worker auto

kevlened
  • 10,846
  • 4
  • 23
  • 17
  • 10
    It is possible to make it work with `pytest-django`? – funnydman Dec 18 '19 at 11:07
  • The fine grained control looks great. Alas, this didn't work for any `airflow` unit tests on my repo. :( – javatarz Jan 23 '20 at 09:07
  • @kevlened is it possible to specify which test to run on which workers? Of all the long tests get assigned to one worker, the tests may still take awhile to complete. – Error - Syntactical Remorse Feb 12 '20 at 17:06
  • @Error-SyntacticalRemorse. It's not currently possible with pytest-parallel, but pytest-xdist may be able to do something like that [with the `--dist` flag](https://stackoverflow.com/a/56595221/1978203) – kevlened Feb 13 '20 at 18:48
  • I love your plugin, would like to leverage your answer to ask you why my test are working with `--workers auto` but not when I use `--workers auto --tests-per-worker 3`? I don't know what it does behind the scene – Federico Piazza Nov 05 '21 at 14:09
  • @kevlened can it handle running two or more different scripts ( files) in parallel? – Bored002 Jun 14 '22 at 08:14
  • Hi @kevlened to be honest a pretty simple selenium + selene + pytest web-testing code can be executed in parallel with no problem using pytest-xdist but does not work out of the box using pytest-parallel using either the `--workers` or the `--tests-per-worker` capabilities. Any ideas of what to follow to resolve these ? Is the project still under maintainance to post an issue in github ? – George Pligoropoulos Aug 11 '22 at 06:22
14

In order to run pytests in parallel you are going to need to install pytest-xdist. Please see the different parallelism tactics listed below, you can use any of those (however I can bet that one of those suits best for your particular case):

Examples

pip install pytest-xdist

# The most primitive case, sending tests to multiple CPUs:
pytest -n NUM

# Execute tests within 3 subprocesses.
pytest --dist=each --tx 3*popen//python=python3.6

# Execute tests in 3 forked subprocess. Won't work on windows.
pytest --dist=each --tx 3*popen//python=python3.6 --boxed

# Sending tests to the ssh slaves
pytest --dist=each --tx ssh=first_slave --tx ssh=seconds_slave --rsyncdir package package

# Sending tests to the socket server, link is available below.
python socketserver.py :8889 &
python socketserver.py :8890 &
pytest --dist=each --tx socket=localhost:8889 --tx socket=localhost:8890

You can provide different values for --dist(-d) parameter, which handles the way tests are distributed across workers, see documentation for more information regarding the usage of --dist.

NOTE: Once tests are executed the socket_server.py is down. I suggest you running socket server from separate terminal windows for debugging purposes

You can introduce more complicated flows, for instance running tests inside docker containers with have started socket servers kind of "pytest workers" and another docker container which communicates to them and serve as "pytest runner".

Andriy Ivaneyko
  • 20,639
  • 6
  • 60
  • 82