111

In order to make sure that the error messages from my module are informative, I would like to see all the error messages caught by assertRaises(). Today I do it for each assertRaises(), but as there are lots of them in the test code it gets very tedious.

How can I print the error messages for all the assertRaises()? I have studied the documentation on http://docs.python.org/library/unittest.html without figuring out how to solve it. Can I somehow monkeypatch the assertRaises() method? I prefer not to change all the assertRaises() lines in the test code, as I most often use the test code the standard way.

I guess this question is related to Python unittest: how do I test the argument in an Exceptions?

This is how I do it today. For example:

#!/usr/bin/env python

def fail():
    raise ValueError('Misspellled errrorr messageee')

And the test code:

#!/usr/bin/env python
import unittest
import failure   

class TestFailureModule(unittest.TestCase):

    def testFail(self):
        self.assertRaises(ValueError, failure.fail)

if __name__ == '__main__':
    unittest.main()  

To check the error message, I simply change the error type in the assertRaises() to for example IOError. Then I can see the error message:

 E
======================================================================
ERROR: testFail (__main__.TestFailureModule)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "test_failure.py", line 8, in testFail
   self.assertRaises(IOError, failure.fail)
  File "/usr/lib/python2.7/unittest/case.py", line 471, in assertRaises
    callableObj(*args, **kwargs)
 File "/home/jonas/Skrivbord/failure.py", line 4, in fail
    raise ValueError('Misspellled errrorr messageee')
ValueError: Misspellled errrorr messageee

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

Any suggestions? /Jonas

EDIT:

With the hints from Robert Rossney I managed to solve the problem. It is not mainly intended for spelling errors, but for making sure that the error messages are really meaningful for the user of the module. The normal functionality of unittest (this is how I use it most of the time) is achieved by setting SHOW_ERROR_MESSAGES = False.

I simply override the assertRaises() method, as seen below. It works like charm!

SHOW_ERROR_MESSAGES = True

class NonexistantError(Exception):
    pass

class ExtendedTestCase(unittest.TestCase):
    def assertRaises(self, excClass, callableObj, *args, **kwargs):
        if SHOW_ERROR_MESSAGES:
            excClass = NonexistantError
        try:
            unittest.TestCase.assertRaises(self, excClass, callableObj, *args, **kwargs)
        except:
            print '\n    ' + repr(sys.exc_info()[1]) 

A fraction of the resulting output:

testNotIntegerInput (__main__.TestCheckRegisteraddress) ... 
    TypeError('The registeraddress must be an integer. Given: 1.0',)

    TypeError("The registeraddress must be an integer. Given: '1'",)

    TypeError('The registeraddress must be an integer. Given: [1]',)

    TypeError('The registeraddress must be an integer. Given: None',)
ok
testCorrectNumberOfBytes (__main__.TestCheckResponseNumberOfBytes) ... ok
testInconsistentLimits (__main__.TestCheckNumerical) ... 
    ValueError('The maxvalue must not be smaller than minvalue. Given: 45 and 47, respectively.',)

    ValueError('The maxvalue must not be smaller than minvalue. Given: 45.0 and 47.0, respectively.',)
ok
testWrongValues (__main__.TestCheckRegisteraddress) ... 
    ValueError('The registeraddress is too small: -1, but minimum value is 0.',)

    ValueError('The registeraddress is too large: 65536, but maximum value is 65535.',)
ok
testTooShortString (__main__.TestCheckResponseWriteData) ... 
    ValueError("The payload is too short: 2, but minimum value is 4. Given: '\\x00X'",)

    ValueError("The payload is too short: 0, but minimum value is 4. Given: ''",)

    ValueError("The writedata is too short: 1, but minimum value is 2. Given: 'X'",)

    ValueError("The writedata is too short: 0, but minimum value is 2. Given: ''",)
ok
testKnownValues (__main__.TestCreateBitPattern) ... ok
testNotIntegerInput (__main__.TestCheckSlaveaddress) ... 
    TypeError('The slaveaddress must be an integer. Given: 1.0',)

    TypeError("The slaveaddress must be an integer. Given: '1'",)

    TypeError('The slaveaddress must be an integer. Given: [1]',)

    TypeError('The slaveaddress must be an integer. Given: None',)
ok
Community
  • 1
  • 1
jonasberg
  • 1,835
  • 2
  • 14
  • 14
  • 5
    Why continue to use assertRaises if you need to check the arguments? Why not simply catch the exception and examine it using `try` and `except`? – S.Lott Dec 29 '11 at 19:59

5 Answers5

174

I once preferred the most excellent answer given above by @Robert Rossney. Nowadays, I prefer to use assertRaises as a context manager (a new capability in unittest2) like so:

with self.assertRaises(TypeError) as cm:
    failure.fail()
self.assertEqual(
    'The registeraddress must be an integer. Given: 1.0',
    str(cm.exception)
)
mkelley33
  • 5,323
  • 10
  • 47
  • 71
  • 2
    NB. assertRaises can be used as a context manager from 'unittest' in Python 2.7. unittest2 backports features for earlier versions of Python. http://docs.python.org/2/library/unittest.html#unittest.TestCase.assertRaises – powlo Mar 26 '14 at 15:52
  • What if , the code fails at the "with" part....in my case , that with part fails...so i want to show a message ..like we can do for other plain asserts e.g self.assertEqual(cm.exception.faultCode, 101001, 'Fault code does not match expected fault code %d' % 101001) – Arindam Roychowdhury Jul 30 '14 at 07:37
  • @arindamroychowdhury, I'm sorry, but I haven't coded any Python in quite some while so I don't know the answer to your question. Best of luck. Perhaps, one of the other people here could better answer your question. Good luck. – mkelley33 Jul 30 '14 at 08:07
  • I am using Python 2.7. I had to replace str(cm.exception) with cm.exception.parameter. – Chuck Sep 12 '16 at 12:35
  • 1
    This answer isn't wrong, but https://stackoverflow.com/a/16282604/6419007 is probably much better. – Eric Duminil Dec 20 '21 at 21:38
80

You are looking for assertRaisesRegex, which is available since Python 3.2. From the docs:

self.assertRaisesRegex(ValueError, "invalid literal for.*XYZ'$",
                       int, 'XYZ')

or:

with self.assertRaisesRegex(ValueError, 'literal'):
    int('XYZ')

PS: if you are using Python 2.7, then the correct method name is assertRaisesRegexp.

rmmariano
  • 896
  • 1
  • 18
  • 32
Iodnas
  • 3,414
  • 24
  • 26
  • yes, but if expected error is not risen, you will never see the message/cannot change the default one. Extremaly uncomfortable when testing some parameters in loop - you don't know for which parameter the function pass through without expected error. – Mesco Jul 07 '17 at 14:02
  • 2
    On python 3.6, it is saying `DeprecationWarning: Please use assertRaisesRegex instead` when using `with self.assertRaisesRegexp( RuntimeError, '...regex...' )` – Evandro Coan Jun 07 '18 at 01:03
  • This is excellent. Concise, very powerful and much more precise than simply testing for some Exception. – Eric Duminil Dec 20 '21 at 21:38
62

Out-of-the-box unittest doesn't do this. If this is something you want to do frequently, you can try something like this:

class ExtendedTestCase(unittest.TestCase):

  def assertRaisesWithMessage(self, msg, func, *args, **kwargs):
    try:
      func(*args, **kwargs)
      self.assertFail()
    except Exception as inst:
      self.assertEqual(inst.message, msg)

Derive your unit test classes from ExtendedTestCase instead of unittest.TestCase.

But really, if you're simply concerned about misspelled error messages, and concerned enough to want to build test cases around it, you shouldn't be inlining messages as string literals. You should do with them what you do with any other important strings: defining them as constants in a module that you import and that someone is responsible for proofreading. A developer who misspells words in his code will also misspell them in his test cases.

Robert Rossney
  • 94,622
  • 24
  • 146
  • 218
  • 27
    +1 for "A developer who misspells words in his code will also misspell them in his test cases." – johnsyweb Dec 29 '11 at 21:54
  • 19
    To me, it's far more egregious when you are testing to see that a _specific_ error is occurring, but a test can 'pass' due to unintended side effects. E.g. the error you expected didn't get raised, but the same error type is raised elsewhere, thus satisfying the test. The test passes, the code is bugged. Same deal for errors that subclass the error you're looking for -- if your test is too general, you end up catching something you don't expect. – Mark Simpson Oct 10 '12 at 11:27
  • 2
    You should use ``inst.args[0]`` instead of ``inst.message`` to run this code both on Python 2 and Python 3 – oblalex Nov 09 '14 at 14:48
  • 4
    This is not true, out of the box unittest does do this. – Jonathan Hartley Feb 04 '15 at 13:20
  • Neato example of subclassing. But I have a hard time believing this is necessary or advantageous, as `unittest`'s 'out of the box' capabilities are quite extensive. – Tom Russell Jun 08 '17 at 06:54
  • @JonathanHartley - I'm also seeing this as the default behavior with unittest in Python3. Any exception which is caught by assertRaises will be magically printed to stdout along with a stacktrace, even if the tested code never logs or prints anything. The only way which I've found works to suppress this is by setting the root logger level to logging.CRITICAL. This is strange because, as stated, nothing is being logged or printed in the test, or the code which it's testing. This workaround also prevents all other logging messages as well, so it's not a practical solution. – Alex Jansen Jun 11 '19 at 00:54
  • @AlexJohnson Strange. Thanks for telling me. I'll go investigate and amend my comments... – Jonathan Hartley Jun 11 '19 at 19:10
43

If you want the error message exactly match something:

with self.assertRaises(ValueError) as error:
  do_something()
self.assertEqual(error.exception.message, 'error message')
yalei du
  • 697
  • 7
  • 13
  • 18
    I had to use `str(error.exception)` instead of `error.exception.message` as `error.exception` has no `message` attribute in my case. – Rik Schoonbeek Feb 26 '20 at 13:54
7

mkelley33 gives nice answer, but this approach can be detected as issue by some code analysis tools like Codacy. The problem is that it doesn't know that assertRaises can be used as context manager and it reports that not all arguments are passed to assertRaises method.

So, I'd like to improve Robert's Rossney answer:

class TestCaseMixin(object):

    def assertRaisesWithMessage(self, exception_type, message, func, *args, **kwargs):
        try:
            func(*args, **kwargs)
        except exception_type as e:
            self.assertEqual(e.args[0], message)
        else:
            self.fail('"{0}" was expected to throw "{1}" exception'
                      .format(func.__name__, exception_type.__name__))

Key differences are:

  1. We test type of exception.
  2. We can run this code both on Python 2 and Python 3 (we call e.args[0] because errors in Py3 don't have message attribute).
oblalex
  • 5,366
  • 2
  • 24
  • 25