1

We're using py.test to execute our integration tests. Since we have quite a lot of tests, we'd like to monitor the progress in a dashboard we use.

Is it possible to configure a webhook or something that pytest will call with the result of each tests executed (passed/failed/skipped)?

I did find the teamcity integration, but we'd prefer to monitor the progress on a different dashboard.

MvdD
  • 22,082
  • 8
  • 65
  • 93
  • Maybe this "fixture-based teardown" will help you https://stackoverflow.com/questions/26405380/how-do-i-correctly-setup-and-teardown-my-pytest-class-with-tests – Damian Jan 15 '19 at 20:05

1 Answers1

1

It depends on what data you want to emit. If a simple completion check will suffice, write a custom pytest_runtest_logfinish hook in the conftest.py file as it directly provides lots of test info:

def pytest_runtest_logfinish(nodeid, location):
    (filename, line, name) = location
    print('finished', nodeid, 'in file', filename,
          'on line', line, 'name', name)

Should you need to access the test result, a custom pytest_runtest_makereport is a good option. You can get the same test info (and more) as above from the item parameter:

import pytest

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield
    result = report.get_result()

    if result.when == 'teardown':
        (filename, line, name) = item.location
        print('finished', item.nodeid, 'with result', result.outcome,
              'in file', filename, 'on line', line, 'name', name)

You can also go with the fixture teardown option, as suggested in the comments:

@pytest.fixture(autouse=True)
def myhook(request):
    yield
    item = request.node
    (filename, line, name) = item.location
    print('finished', item.nodeid, 'in file', filename,
          'on line', line, 'name', name)

However, it depends on when you want your webhook to emit - the custom hookimpls above will run when the test finished and all fixtures have finalized, while with the fixture example, you can't guarantee that all fixtures have finalized yet as there's no explicit fixture ordering. Also, should you need the test result, you can't access it in a fixture.

hoefling
  • 59,418
  • 12
  • 147
  • 194
  • Yes, I'd like to get access to the test result. The `pytest_runtest_makereport` looks promising. Let me test that out. Thanks! – MvdD Jan 15 '19 at 22:47
  • The `pytest_runtest_makereport` method works, but the results is 'passed' when `result.when == 'teardown'`. I can see the result being 'failed' when `result.when == 'call'`. – MvdD Jan 22 '19 at 01:10