How to create global error handler in a multi-threaded python applcation

Question

I am developing a multi-threaded application in python. I have following scenario.

1. There are 2-3 producer threads which communicate with DB and get some data in large chunks and fill them up in a queue
2. There is an intermediate worker which breaks large chunks fetched by producer threads into smaller ones and fill them up in another queue.
3. There are 5 consumer threads which consume queue created by intermediate worker thread.
4. objects of data sources are accessed by producer threads through their API. these data sources are completely separate. So these producer understands only presence or absence of data which is supposed to be given out by data source object.
5. I create threads of these three types and i make main thread wait for completion of these threads by calling join() on them.

Now for such a setup I want a common error handler which senses failure of any thread, any exception and decides what to do. For e.g if I press ctrl+c after I start my application, main thread dies but producer, consumer threads continue to run. I would like that once ctrl+c is pressed entire application should shut down. Similarly if some DB error occurs in data source module, then producer thread should get notified of that.

This is what I have done so far:

I have created a class ThreadManager, it's object is passed to all threads. I have written an error handler method and passed it to sys.excepthook. This handler should catch exceptions, error and then it should call methods of ThreadManager class to control the running threads. Here is snippet:

class Producer(threading.Thread):
    ....
    def produce():
        data = dataSource.getData()

class DataSource:
    ....
    def getData():
        raise Exception("critical")

def customHandler(exceptionType, value, stackTrace):
     print "In custom handler"

sys.excepthook = customHandler

Now when a thread of producer class calls getData() of DataSource class, exception is thrown. But this exception is never caught by my customHandler method.

What am I missing? Also in such scenario what other strategy can I apply? Please help. Thank you for having enough patience to read all this :)

Answer 1

What you need is a decorator. In essence you are modifying your original function and putting in inside a try-except:

def exception_decorator(func):
    def _function(*args):
        try:
            result = func(*args)
        except:
            print('*** ESC default handler ***')
            os._exit(1)
        return result
    return _function

If your thread function is called myfunc, then you add the following line above your function definition

@exception_decorator
def myfunc():
    pass;

Answer 2

Can't you just catch "KeyboardInterrupt" when pressing Ctrl+C and do:

for thread in threading.enumerate():
    thread._Thread__stop()
    thread._Thread__delete()
while len(threading.enumerate()) > 1:
    time.sleep(1)
os._exit(0)

and have a flag in each threaded class which is self.alive you could theoretically call thread.alive = False and have it stop gracefully?

for thread in threading.enumerate():
    thread.alive = False
    time.sleep(5) # Grace period
    thread._Thread__stop()
    thread._Thread__delete()
while len(threading.enumerate()) > 1:
    time.sleep(1)
os._exit(0)

example:

import os
from threading import *
from time import sleep

class worker(Thread):
    def __init__(self):
        self.alive = True
        Thread.__init__(self)
        self.start()
    def run(self):
        while self.alive:
            sleep(0.1)

runner = worker()

try:
    raw_input('Press ctrl+c!')
except:
    pass
for thread in enumerate():
    thread.alive = False
    sleep(1)
    try:
        thread._Thread__stop()
        thread._Thread__delete()
    except:
        pass
# There will always be 1 thread alive and that's the __main__ thread.
while len(enumerate()) > 1:
    sleep(1)
os._exit(0)

Try going about it by changing the internal system exception handler?

import sys
origExcepthook = sys.excepthook
def uberexcept(exctype, value, traceback):
    if exctype == KeyboardInterrupt:
        print "Gracefully shutting down all the threads"
        # enumerate() thingie here.
    else:
        origExcepthook(exctype, value, traceback)
sys.excepthook = uberexcept