how can assert exception message with pytest xfail (assertion on exception type is working)

Question

Let's assume I have a file test_scratch_2.py


import pytest


def my_fun(number: int):
    if number == 1:
        raise ValueError("Some Message for number 1")

    if number == 2:
        raise ValueError("Some Message for number 2")

    if number == 3:
        return int("number")

    return number ** 2


@pytest.mark.parametrize("digit", [
    # example 1
    pytest.param(1, id="first",
                 marks=pytest.mark.xfail(raises=ValueError,
                                         strict=True,
                                         reason="Some Message for number 1")),
    # example 2
    pytest.param(2, id="second",
                 marks=pytest.mark.xfail(raises=ValueError,
                                         strict=True,
                                         reason="Some Message for number 1")),
    pytest.param(3, id="third",
                 marks=pytest.mark.xfail(raises=ValueError,
                                         strict=True,
                                         reason="Some Message for number xxxxxxxxx")),

    4,
    5,
    60000
])
def test_my_fun(digit):
    assert my_fun(digit) > 0

And I run the tests

> pytest test_scratch_2.py -vvv                                                                                                                                                                                                
======================= test session starts ============================
platform linux -- Python 3.7.5, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 --.../bin/python3
cachedir: .pytest_cache
rootdir:....config/JetBrains/PyCharm2020.1/scratches
collected 6 items                                                                                                                                                                                                             

test_scratch_2.py::test_my_fun[first] XFAIL                                                                                                                                                                             [ 16%]
test_scratch_2.py::test_my_fun[second] XFAIL                                                                                                                                                                            [ 33%]
test_scratch_2.py::test_my_fun[third] XFAIL                                                                                                                                                                             [ 50%]
test_scratch_2.py::test_my_fun[4] PASSED                                                                                                                                                                                [ 66%]
test_scratch_2.py::test_my_fun[5] PASSED                                                                                                                                                                              [ 83%]
test_scratch_2.py::test_my_fun[60000] PASSED

There is a bug in case of test_scratch_2.py::test_my_fun[third] XFAIL.

It fails with an expected exception but not with the expected reason.

All assertions are based on exception type and not on exception messages.

How I could check the exception message?

Simon David · Answer 1 · 2023-08-29T01:20:40.987

You can use pytest's with pytest.raises(ErrorWhichMustBeRaised, match='Error message which must appear'): context manager to achieve the desired behaviour, as suggested in this SO answer to a similar question.

To make this work, I divided your tests into "happy path" tests and "exceptions" tests. Please note that your use of xfail is technically an anti-pattern: "It is better to use the pytest.mark.xfail marker when possible to declare a test to be xfailed under certain conditions like known bugs or missing features." (source: official doc)

Testing the code below (in quiet mode) returns the error message you would want in your case:

============================================================== short test summary info ===============================================================
FAILED main.py::test_my_fun_exceptions[second] - AssertionError: Regex pattern did not match.
FAILED main.py::test_my_fun_exceptions[third] - AssertionError: Regex pattern did not match.
2 failed, 4 passed in 0.11s

Please note that id='third' fails as well. That's because attempting to convert a string to an integer will raise a ValueError with a different error message than the one which appears in the code.

Code (python 3.11.4):

import pytest


def my_fun(number: int):
    if number == 1:
        raise ValueError("Some Message for number 1")

    if number == 2:
        raise ValueError("Some Message for number 2")

    if number == 3:
        return int("number")

    return number**2


@pytest.mark.parametrize("digit", [4, 5, 60000])
def test_my_fun(digit):
    assert my_fun(digit) > 0


@pytest.mark.parametrize(
    "digit, error, match_str",
    [
        # example 1
        (pytest.param(1, ValueError, "Some Message for number 1", id="first")),
        # example 2
        (pytest.param(2, ValueError, "Some Message for number 1", id="second")
         ),
        # example 3
        (pytest.param(
            3, ValueError, "Some Message for number xxxxxxxxx", id="third"))
    ])
def test_my_fun_exceptions(digit, error, match_str):
    with pytest.raises(error, match=match_str):
        assert my_fun(digit) > 0

yeah, reason is not match=exception message – Andrei.Danciuc Aug 30 '23 at 06:44 — Andrei.Danciuc, Aug 30 '23 at 06:44

score 1 · Answer 2 · answered Aug 30 '23 at 07:00

Thanks to Simon I realized that it is really an anti pattern

Splitting test into success and fail is questionable ...

Here bellow I provide a solution without split.


from contextlib import nullcontext as does_not_raise
import pytest


def my_fun(number: int):
    if number == 1:
        raise ValueError("Some Message for number 1")

    if number == 2:
        raise ValueError("Some Message for number 2")

    if number == 3:
        return int("number")

    return number ** 2


@pytest.mark.parametrize("digit, expect_to_raise", [
    pytest.param(1, pytest.raises(ValueError, match="Some Message for number 1"), id="first"),
    pytest.param(2, pytest.raises(ValueError, match="Some Message for number 2"), id="second"),
    pytest.param(3, pytest.raises(ValueError, match=r"invalid literal for int\(\) with base 10: 'number'"), id="third"),
    pytest.param(4, does_not_raise(), id="fourth"),
])
def test_my_fun(digit, expect_to_raise):
    with expect_to_raise:
        assert my_fun(digit) > 0

and the output

============================= test session starts ==============================
collecting ... collected 4 items

test_a.py::test_my_fun[first] PASSED                                     [ 25%]
test_a.py::test_my_fun[second] PASSED                                    [ 50%]
test_a.py::test_my_fun[third] PASSED                                     [ 75%]
test_a.py::test_my_fun[fourth] PASSED                                    [100%]

============================== 4 passed in 0.01s ===============================

how can assert exception message with pytest xfail (assertion on exception type is working)

2 Answers2