Pytest checking messages returned by errors

Question

Something about the interaction between pytest, str() and python Error types breaks full error testing where we want to confirm the EXACT TEXT returned by the error.

Example below:

def erroring_func(name, required_item_list):
   # skip boring bit. Just throw an error.
   raise KeyError(f'{name} is missing required item(s): {required_item_list')

def test_erroring_func():
    with pytest.raises(KeyError) as err:
        name = 'This dataframe'
        required_item_list = ['a column']
        _ = erroring_func(name, required_item_list)
    assert str(err.value) == f"{name} is missing required item(s): {required_item_list}"

This looks sensible, but will return the error:

assert '"This dataframe is missing required item(s): [\'lat\']"' == "This dataframe is missing required item(s): ['lat']

Somehow, str(err.value) creates single backslashes in the output that are EXTREMELY difficult to recreate in an f-string (actually impossible) or to insert in a string once created.

so, the problem is `KeyError` has a custom `__str__` and the constructor for `KeyError` is meant to be the missing key, that's why you see it quoted. the backslashes aren't actually there, it's just the representation of the string starts with a single quote so embedded single quotes would be displayed that way in code. `KeyError` is probably the wrong exception type as well — anthony sottile, Jul 23 '21 at 01:31
The strings in the assert don't match so the `__str__` method defined for `KeyError` has certainly altered something. I'm tempted to believe the actual error log that it is the backslashes but it could be something else? As for why KeyError, this is a custom check to confirm df columns before expensive calculations and thus bringing forward what would later be a key error (and making the error message actually useful). Could make a custom error class, but am interesting in understanding why the default class plays some poorly with pytest. — Mark_Anderson, Jul 23 '21 at 16:51
change your assertion to use `'"message"'` as KeyError does -- notice the quoting is *both* types of quotes (the repr of the string you've passed in). compare `str(KeyError("foo'bar"))` and `str(ValueError("foo'bar"))` and you'll see better what `KeyError`'s `__str__` is doing — anthony sottile, Jul 23 '21 at 16:55
I'd also strongly recommend *against using logic in tests* -- it's obscuring your problem and confusing you: https://testing.googleblog.com/2014/07/testing-on-toilet-dont-put-logic-in.html — anthony sottile, Jul 23 '21 at 16:57
Thanks for the help! Needed to double wrap text strings in quotes of approriate type and order (yuck). Google thinkpiece is interesting one, thanks. Overall I disagree because that thinking makes unit tests very hard to maintain, although I can see that a larger organisation with slower changing specs might think that way (ie. if you don't change unit tests, who cares about maintainability). As per one response to it: https://www.andrewtrumper.com/2014/08/complex-logic-in-unit-tests.html — Mark_Anderson, Jul 23 '21 at 17:51
This code does not make sense and the problem doesn't make much sense either. The built-in `KeyError` does not have a `value` attribute. Please see [mre]. — Karl Knechtel, Aug 07 '22 at 05:09
Karl, you don't know pytest well enough to comment. `err.value` is standard testing https://docs.pytest.org/en/7.1.x/how-to/assert.html. Question has MRE, which you can verify by running the code. However, the solutions presented are now out of date so I'll post an update later — Mark_Anderson, Aug 09 '22 at 13:36

Mark_Anderson · Answer 1 · 2022-08-09T15:05:31.083

An incomplete patch (missing the major value of verbose errors) is to test that a fixed substring exists in the returned error.

def test_erroring_func()
    with pytest.raises(KeyError) as err:
        name = 'This dataframe'
        required_item_list = ['a column']
        _ = erroring_func(name, required_item_list)
    assert "is missing required item(s):" in str(err.value)

PS. Updating to more modern pytest syntax, a regex match can be defined as a arg in pytest.raises

def test_erroring_func():
    with pytest.raises(KeyError, match="is missing required item(s):") as err:
        name = 'This dataframe'
        required_item_list = ['a column']
        _ = erroring_func(name, required_item_list)

Mark_Anderson · Accepted Answer · 2022-08-09T15:02:59.800

0

You can completely solve by matching how KeyError alters text. This can be done with an f-string with single quotes and then double quotes f'"your text {here}"'

assert str(err.value) == f'"{name} is missing required item(s): {required_item_list}"'

(With thanks to Anthony Sotile)

edited Aug 09 '22 at 15:02

answered Jul 23 '21 at 17:56

Mark_Anderson

1,229
1
12
34

Pytest checking messages returned by errors

2 Answers2