How to aggregate test results to publish to testrail after executing pytest tests with xdist?

Question

I'm running into a problem like this. I'm currently using pytest to run test cases, and reducing execution time using xdist to run tests in parallel and publishing tests results to TestRail. The issue is when using xdist, pytest-testrail plugin creates Test-Run for each xdist workers and then publishes test cases like Untested.

I tried this hook pytest_terminal_summary to prevent pytest_sessionfinish plugin hook from being call multiple times.

I expect only one test run is created, but still multiple test runs are created.

https://github.com/allankp/pytest-testrail/blob/master/pytest_testrail/plugin.py#L196 I replaced `pytest_terminal_summary` plugin hook with `pytest_sessionfinish` plugin hook — Thành Nguyen, Jul 18 '19 at 15:20
@ThànhNguyen hey did you find a solution for this problem ? can you provide some hints. — virulent-slash, Apr 25 '21 at 10:37
hey @virulent-slash I followed this https://github.com/allankp/pytest-testrail/pull/90#issuecomment-469741084 — Thành Nguyen, Apr 26 '21 at 11:32

score 2 · Answer 1 · answered Aug 21 '19 at 14:25

I ran in to the same problem, but found a kind of workaround with duct tape. I found that all results are collecting properly in test run, if we run the tests with --tr-run-id key. If you are using jenkins jobs to automate processes, you can do following: 1) create testrun using testrail API 2) get ID of this test run 3) run the tests with --tr-run-id=$TEST_RUN_ID

I used these docs: http://docs.gurock.com/testrail-api2/bindings-python http://docs.gurock.com/testrail-api2/reference-runs

from testrail import *
import sys

client = APIClient('URL')
client.user = 'login'
client.password = 'password'

result = client.send_post('add_run/1', {"name": sys.argv[1], "assignedto_id": 1}).get("id")
print(result)

then in jenkins shell

RUN_ID=`python3 testrail_run.py $BUILD_TAG`

and then

python3 -m pytest -n 3 --testrail --tr-run-id=$RUN_ID --tr-config=testrail.cfg ...

Thanks @Sergei. I found out the way to do it with parsing the tests results from xml and the help of conftest.py — Thành Nguyen, Sep 11 '19 at 08:55

score 0 · Answer 2 · answered Aug 01 '23 at 09:13

0

Now you can use the TRCLI (TestRail CLI) to get around this problem and avoid unnecessary code. Please refer this answer for details - https://stackoverflow.com/a/76809919/1502300

answered Aug 01 '23 at 09:13

virulent-slash

66
7

How to aggregate test results to publish to testrail after executing pytest tests with xdist?

2 Answers2