Catch an exception raised by a child process in a parent process

Question

I'm trying to catch an Exception raised by a child process, and have been running into some issues. The gist of my code is:

class CustomException(Exception):
  def __init__(self, msg):
    self.msg = msg

  def __str__(self):
    return self.msg    

def update(partition):
  if os.getpid() % 2 == 0:
    raise CustomException('PID was divisible by 2!')
  else:
    # Do something fancy

if __name__ == '__main__':
  try:
    some_response = get_response_from_another_method()
    partition_size = 100
    p = Pool(config.NUMBER_OF_PROCESSES)
    for i in range(0, NUMBER_OF_PROCESSES):
      partition = get_partition(some_response, partition_size)
      x = p.apply_async(update, args=(partition,))
    x.get()
    p.close()
    p.join()
  except CustomException as e:
    log.error('There was an error')
    if email_notifier.send_notification(e.msg):
      log.debug('Email notification sent')
    else:
      log.error('An error occurred while sending an email.')

When I run this, I am seeing:

  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class 'CustomException'>, ())

Is there some facility to do this? Thanks!!

jme · Answer 1 · 2015-11-19T02:16:04.040

In short, this is something of a quirk in Python 2, and a related issue is referenced in this bug report. It has to do with how exceptions are pickled. The simplest solution is perhaps to alter CustomException so that it calls its parent class initializer. Alternatively, if you're able, I'd suggest moving to Python 3.

For example, this code works fine in both Python 2 and Python 3:

from multiprocessing import Pool

class CustomException(Exception):
    pass

def foo():
    raise CustomException('PID was divisible by 2!')

pool = Pool()
result = pool.apply_async(foo, [])

But if we alter CustomException so that it has a required argument:

class CustomException(Exception):
    def __init__(self, required):
        self.required = required

The above example results in a TypeError being raised under Python 2. It works under Python 3.

The problem is that CustomException inherits Exception's __reduce__ method, which tells Python how to pickle an instance. The inherited __reduce__ knows nothing about CustomException's call signature, so unpickling isn't done correctly.

A quick fix is to simply call the parent class's __init__:

class CustomException(Exception):
    def __init__(self, msg):
        super(Exception, self).__init__()
        self.msg = msg

But since you really aren't doing anything special with the message, why not just define:

class CustomException(Exception):
    pass

Catch an exception raised by a child process in a parent process

1 Answers1