Catching unpickleable exceptions and re-raising

Question

This is a followup to my question Hang in Python script using SQLAlchemy and multiprocessing. As discussed in that question, pickling exceptions is problematic in Python. This is usually not a issue, but one case when it is, is when errors occur in the python multiprocessing module. Since multiprocessing moves objects around by pickling, if an error occurs inside a multiprocessing process, the entire process may hang, as demonstrated in that question.

One possible approach is to fix all the problematic exceptions, as discussed in that question. This is not easy, since one cannot easily know in advance which exceptions may be called. An alternative approach, which was suggested by lbolla in an answer to the question, is to catch the exception, construct an equivalent harmless exception, and then rethrow. However, I'm not sure of exactly how to do this. Consider the following code.

class BadExc(Exception):
    def __init__(self, message, a):
        '''Non-optional param in the constructor.'''
        Exception.__init__(self, message)
        self.a = a

import sys
try:
    try:
        #print foo
        raise BadExc("bad exception error message", "a")
    except Exception, e:
        raise Exception(e.__class__.__name__ + ": " +str(e)), None, sys.exc_info()[2]
except Exception, f:
    pass

import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]

This code pickles and unpickles the exception, and then throws it, giving the error

raising error
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
Exception: BadExc: bad exception error message

Credits to Glenn Maynard's answer to "“Inner exception” (with traceback) in Python?". This has the important stuff, namely the traceback, the error message, and the exception type, so this might be the best one can do. But ideally I'd like something that looks exactly like the original exception, namely

Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
__main__.BadExc: bad exception error message

or more generally, with the name of the exception in the front, rather than Exception. Is this possible?

Alternatively, instead of the BadExc class, one can use the print foo statement instead, which gives a NameError. However, this exception does not require special handling.

Answer 1

You can override sys.excepthook to achieve what you want. It at least works for this example, but it's pretty hacky so please test and no promises :-)

import sys

def excepthook_wrapper(type, value, traceback):
    if len(value.args) == 2:
        name, msg = value.args
        value.args = (msg,)
        sys.__excepthook__(name, value, traceback)
    else:
        sys.__excepthook__(type, value, traceback)

sys.excepthook = excepthook_wrapper

(Edit: I'm not really happy with this because now 'normal' Exceptions with two arguments will get handled differently too. Possible solution, 'tag' your special Exceptions by passing "PICKLED" as a first argument and then check for that, instead of checking for the length of the args.)

And then create the Exception with two arguments, the name (__module__.__class__) and the Exception message (str(e)):

try:
    try:
        #print foo
        raise BadExc("bad exception error message", "a")
    except Exception, e:
        cls = e.__class__
        if hasattr(cls, '__module__'):
            name = '{0}.{1}'.format(cls.__module__, cls.__name__)
        else:
            name = cls.__name__
        raise Exception(name, str(e)), None, sys.exc_info()[2]
except Exception, f:
    pass

Then this:

import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]

Prints:

raising error
Traceback (most recent call last):
  File "test.py", line 18, in <module>
    raise BadExc("bad exception error message", "a")
__main__.BadExc: bad exception error message