How can I catch an exception thrown in a function that I'm calling dynamically?

Question

I got distracted and ended up writing a test framework in python. I'm struggling with a particular problem that this throws up.

During my assertion, I want to throw an exception as a way of bubbling a problem up to the run_test() method without requiring the user to have any knowledge of the framework. The problem is that when I do that, it seems that the try/catch block is not honoured.

Here is a cut down version of my fledgling framework:

# test_framework.py
import inspect
import module_containing_tests

class AssertionException(Exception):
    def __init__(self, message):
        self.message = message
    def __str__(self):
        return self.message

def run_test(test_name, test_method):
    try:
        print(">", test_name)
        test_method()
        print("Passed")
    except AssertionException as error:
        print("Failed")
        print(str(error))

def assert_true(conditional):
    if not conditional:
        raise AssertionException("Expected True. Was False")

def test(func):
    func.is_test = True
    return func

members = inspect.getmembers(module_containing_tests)
for member in members:
    if "is_test" in dir(member[1]) and not member[0] == "module_containing_tests":
        run_test(member[0], member[1])

The module containing the tests will look like this:

# module_containing_tests.py
from test_framework import *

@test
def passing_test():
    assert_true(1 + 2 == 3)

@test
def failing_test():
    assert_true(1 + 2 == 5)

The output has all the exception stack tracing in it and it also halts the execution

λ python test_framework.py
> failing_test
Traceback (most recent call last):
  File "test_framework.py", line 29, in <module>
    run_test(member[0], member[1])
  File "test_framework.py", line 13, in run_test
    test_method()
  File "C:\Git\simpy-test\module_containing_tests.py", line 9, in failing_test
    assert_true(1 + 2 == 5)
  File "C:\Git\simpy-test\test_framework.py", line 20, in assert_true
    raise AssertionException("Expected True. Was False")
test_framework.AssertionException: Expected True. Was False

What I want is something like this:

λ python test_framework.py
> failing_test
Expected True. Was False
Failed
> passing_test
Passed

But test method is invoked within a try/catch block. Surely any unhandled exceptions within that block will be caught by it? — cohen990, Feb 02 '18 at 03:37

Debo · Accepted Answer · 2018-02-04T12:21:36.603

I think the issue is partly in the circular reference between the two files that might mess up with the visibility of the methods (as somehow explained here) and partly, maybe, in the approach. If you think about how many other testing framework work, you often have 3 elements, the unit to test, the testing framework and a test runner.

So if we try to split everythig folllowing that logic you end up having:

test_framework.py

# test_framework.py
class AssertionException(Exception):
    pass


def test(f):
   f.is_test = True
   return f


def assert_true(conditional):
    if not conditional:
        raise AssertionException("Expected True. Was False")

test_runner.py

# test_runner.py
import inspect

import unit_test
from test_framework import AssertionException


def run_test(test_name, test_method):
    try:
        print(">", test_name)
        test_method()
        print("Passed")
    except AssertionException as error:
        print("Failed with AssertionException: " + str(error))
    except Exception as error:
        print("Failed with Exception: " + str(error))


if __name__ == "__main__":
    members = inspect.getmembers(unit_test)
    for member in members:
        if "is_test" in dir(member[1]):
            run_test(member[0], member[1])

unit_test.py

# unit_test.py
from test_framework import *


@test
def a_passing_test():
    assert_true(1 + 2 == 3)


@test
def z_failing_test():
    assert_true(1 + 2 == 5)

With this setup the circular dependency is removed and all the visibility context are respected and the output/behaviour is the expected one.

I hope it helps.

score 1 · Answer 2 · answered Feb 02 '18 at 19:19

Not sure this is what you want but this works.
Copied from here Hide traceback unless a debug flag is set

Output:

$ ./test_framework.py
> a_passing_test
Passed
> z_failing_test
test_framework.AssertionException: Expected True. Was False

First file:

#!/usr/bin/env python3
#test_framework.py

import inspect 
import module_containing_tests 
import sys


class AssertionException(Exception):
    def __init__(self, message):
        self.message = message
    def __str__(self):
        return self.message

def run_test(test_name, test_method):
    try:
        print(">", test_name)
        test_method()
        print("Passed")
    except AssertionException as error:
        print("Failed")
        print(str(error))

def assert_true(conditional):
    if not conditional:
        raise AssertionException("Expected True. Was False")

def test(func):
    func.is_test = True
    return func


sys.tracebacklimit=0
members = inspect.getmembers(module_containing_tests)
for member in members:
    if "is_test" in dir(member[1]) and not member[0] == "module_containing_tests":
        run_test(member[0], member[1])

second file:

#!/usr/bin/env python3
#module_containing_tests.py
from test_framework import *

@test
def a_passing_test():
    assert_true(1 + 2 == 3)

@test
def z_failing_test():
    assert_true(1 + 2 == 5)

How can I catch an exception thrown in a function that I'm calling dynamically?

2 Answers2