Customizing error message for specific exceptions in pytest

Question

I'm trying to write a pytest plugin to customize the appearance of specific exceptions - more specifically, mocking exceptions (method expected to be called was not called etc.), because there's a lot of useless noise in the traceback of those exceptions.

This is what I've got so far, which works, but is extremely hacky:

import pytest
import flexmock

@pytest.hookimpl()
def pytest_exception_interact(node, call, report):
    exc_type = call.excinfo.type

    if exc_type == flexmock.MethodCallError:
        entry = report.longrepr.reprtraceback.reprentries[-1]
        entry.style = 'short'
        entry.lines = [entry.lines[-1]]
        report.longrepr.reprtraceback.reprentries = [entry]

I think I'm doing the right thing with the hookimpl and checking the exception type with a simple if statement.

I tried replaceing report.longrepr with a simple string, which also worked, but then I lose out on formatting (colors in the terminal).

As an example of the type of output I want to shorten, here's a mock assertion failure:

=================================== FAILURES ====================================
_______________________ test_session_calls_remote_client ________________________

    def test_session_calls_remote_client():
        remote_client = mock.Mock()
        session = _make_session(remote_client)
        session.connect()
        remote_client.connect.assert_called_once_with()
        session.run_action('asdf')
>       remote_client.run_action.assert_called_once_with('asdff')

tests/unit/executor/remote_test.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/python-3.6.3/lib/python3.6/unittest/mock.py:825: in assert_called_once_with
    return self.assert_called_with(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_mock_self = <Mock name='mock.run_action' id='139987553103944'>
args = ('asdff',), kwargs = {}, expected = (('asdff',), {})
_error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x7f51646269d8>
actual = call('asdf'), cause = None

    def assert_called_with(_mock_self, *args, **kwargs):
        """assert that the mock was called with the specified arguments.

            Raises an AssertionError if the args and keyword args passed in are
            different to the last call to the mock."""
        self = _mock_self
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            raise AssertionError('Expected call: %s\nNot called' % (expected,))

        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher((args, kwargs))
        actual = self._call_matcher(self.call_args)
        if expected != actual:
            cause = expected if isinstance(expected, Exception) else None
>           raise AssertionError(_error_message()) from cause
E           AssertionError: Expected call: run_action('asdff')
E           Actual call: run_action('asdf')

/opt/python-3.6.3/lib/python3.6/unittest/mock.py:814: AssertionError
====================== 1 failed, 30 passed in 0.28 seconds ======================

I don't really get what you want to achieve. Do you want to reduce the depth of the stack or to remove specific calls? Could you give an example of what you get and what you wish to achieve? — Thomas Moreau, Mar 16 '17 at 23:25
The main problem I want to solve is that using using python's `mock`/`unittest.mock`/`flexmock` libraries, when mock expectations fail, there's a huge stack trace cluttering up the screen, when "method X was not called with arguments Y" would suffice. — Andreas, Mar 26 '17 at 20:41
Do you have an example of what the output currently looks like, and what you want it to look like? — theY4Kman, May 04 '17 at 19:38
As others have said, it would help immensely if you had an example of how it looks right now (the output you dislike), and an example of how you would like the output to look instead. — exhuma, May 22 '17 at 11:00

score 1 · Answer 1 · edited May 15 '18 at 18:00

If your goal is to make the stacktrace easier to read, then you can use the below code block to output a custom error message. This custom error message appears at end of the stacktrace, so you don't need to scroll up:

with raises(ZeroDivisionError, message="Expecting ZeroDivisionError"):
    pass
--> Failed: Expecting ZeroDivisionError

Source: pytest's documentation. So, instead of making a plugin you can pipe the output of pytest through something like grep and filter out the useless parts of the stacktrace.

Based on what I read in the documentation you're using the correct pytest decorator and hook function(pytest_exception_interact). But, type checking for the error might be unnecessary. This section of the documentation says that "This hook is only called if an exception was raised that is not an internal exception like skip.Exception."

score 0 · Answer 2 · answered May 15 '18 at 10:48

I achieve a similar goal (custom error reporting from pytest) with a similar approach to the one proposed by Mikaeil -- intercepting the output, parsing it, then applying any filters that I need.

Here are the steps:

I run pytest from a script so I can control the execution context and configuration so it is consistent.
I use this to configure pytest to output errors in json format to a log file.
I also redirect stdout and stderr to another log file (to hide it).
If pytest returns a nonzero result, I parse the json file and pass the results to a 'build monitor' class, which then outputs to stdout and to HTML.

I use the same build monitor class for all of my static analysis tools, unit tests, C compiler etc... so the build process can report errors in a nice to look at (HTML and console) and uniform way.

The relevant file for pytest is here: https://github.com/wtpayne/hiai/blob/master/a3_src/h70_internal/da/check/pytest.py

There is also a pytest plugin to control coverage reporting here: https://github.com/wtpayne/hiai/blob/master/a3_src/h70_internal/da/check/pytest_da.py

And the output formatting class is here: https://github.com/wtpayne/hiai/blob/master/a3_src/h70_internal/da/monitor/console_reporter.py

Customizing error message for specific exceptions in pytest

2 Answers2

Linked