continue test after assertion fail using xmlrunner and unittest in python

Question

I am writing a simple test using XMLrunner and unittest in python. When using assert the test fails and does not continue. I want the test to continue to the end and then fail. Is it possible? I will attach a very simple code demostrating what I need to do.

import xmlrunner
import unittest

class TestExp(unittest.TestCase):

    def setUp(self):
        self.list = range(1,10)

    def test_example(self):
        for i in self.list:
            self.assertTrue(i == 3, str(i) + "message")


if __name__ == '__main__':
    unittest.main(
        testRunner=xmlrunner.XMLTestRunner(output='test-reports'),
        failfast=False, buffer=False, catchbreak=False)

as an output an XML is generated, however containing only the first failed assertion, I need to run the rest of the assertions and also generate test-reports from them, when using try/except i cannot see the testcase fail in XML file.

<?xml version="1.0" ?>
<testsuite errors="1" failures="0" name="TestExp-20150429152621" tests="1" time="0.000">
    <testcase classname="TestExp" name="test_example" time="0.000">
        <error message="1message" type="AssertionError">
<![CDATA[Traceback (most recent call last):
  File "xmlrunsample.py", line 14, in test_example
    self.assertTrue(i == 3, str(i) + "message")
AssertionError: 1message
]]>     </error>
    </testcase>
    <system-out>
<![CDATA[]]>    </system-out>
    <system-err>
<![CDATA[]]>    </system-err>
</testsuite>

This is what i get as test-report output, containing only 1 assertion fail, how can i make the script continue to assert the rest of test cases?

score 2 · Accepted Answer · edited May 23 '17 at 12:22

2

You are structuring your tests the wrong way. The test runner sees this as a single test. When it catches an assertion, the test failes.

If you want to stay close to your code, you have to catch the assertions yourself, and re-throw one in the end. This is obviously smelly code, as it makes the origin of the fail opaque.

Ideally you'd have to re-engineer your tests. If you have assertions which are independent of each other (i.e. you are interested in the next one, even if the previous failed), you have independent test cases. The Testrunner is responsible for iteration through your tests, and it will catch each assertion and output it.

Look at the fine documentation on how to do it.

If you are looking for parametrized test cases, here on SO is an idea, on which you can start.

edited May 23 '17 at 12:22

Community

1
1

answered Apr 29 '15 at 14:04

knitti

6,817
31
42

not sure i understand, its the first time i am doing this, could u please give me more details or a hint on how to do it? thx – Jan Galik Apr 29 '15 at 14:07
Lets say i want to test numbers 1 through 1000 and test every one if it is even. Before i just tested it with a condition and appended the bug to an array if the condition returned false. But now I need to do the same thing, but with all the odd numbers listed as errors in XML test report file. – Jan Galik Apr 29 '15 at 14:20
I need the proper JUnit XML output, because this test should run automatically in JIRA(of course not exactly this test, but the problem i have is the same) – Jan Galik Apr 29 '15 at 14:23
you need parametrized tests (look at the link), e.g. a generator for a number of alike tests – knitti Apr 29 '15 at 14:25

continue test after assertion fail using xmlrunner and unittest in python

1 Answers1