Interruptible thread join in Python

Question

Is there any way to wait for termination of a thread, but still intercept signals?

Consider the following C program:

#include <signal.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <stdlib.h>

void* server_thread(void* dummy) {
    sleep(10);
    printf("Served\n");
    return NULL;
}

void* kill_thread(void* dummy) {
    sleep(1); // Let the main thread join
    printf("Killing\n");
    kill(getpid(), SIGUSR1);
    return NULL;
}

void handler(int signum) {
    printf("Handling %d\n", signum);
    exit(42);
}

int main() {
    pthread_t servth;
    pthread_t killth;

    signal(SIGUSR1, handler);

    pthread_create(&servth, NULL, server_thread, NULL);
    pthread_create(&killth, NULL, kill_thread, NULL);

    pthread_join(servth, NULL);

    printf("Main thread finished\n");
    return 0;
}

It ends after one second and prints:

Killing
Handling 10

In contrast, here's my attempt to write it in Python:

#!/usr/bin/env python
import signal, time, threading, os, sys

def handler(signum, frame):
    print("Handling " + str(signum) + ", frame:" + str(frame))
    exit(42)
signal.signal(signal.SIGUSR1, handler)

def server_thread():
    time.sleep(10)
    print("Served")
servth = threading.Thread(target=server_thread)
servth.start()

def kill_thread():
    time.sleep(1) # Let the main thread join
    print("Killing")
    os.kill(os.getpid(), signal.SIGUSR1)
killth = threading.Thread(target=kill_thread)
killth.start()

servth.join()

print("Main thread finished")

It prints:

Killing
Served
Handling 10, frame:<frame object at 0x12649c0>

How do I make it behave like the C version?

Answer 1

Threads in Python are somewhat strange beasts given the global interpreter lock. You may not be able to achieve what you want without resorting to a join timeout and isAlive as eliben suggests.

There are two spots in the docs that give the reason for this (and possibly more).

The first:

From http://docs.python.org/library/signal.html#module-signal:

Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), pause(), setitimer() or getitimer(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.

The second, from http://docs.python.org/library/thread.html#module-thread:

Threads interact strangely with interrupts: the KeyboardInterrupt exception will be received by an arbitrary thread. (When the signal module is available, interrupts always go to the main thread.)

EDIT: There was a decent discussion of the mechanics of this on the python bug tracker here: http://bugs.python.org/issue1167930. Of course, it ends with Guido saying: " That's unlikely to go away, so you'll just have to live with this. As you've discovered, specifying a timeout solves the issue (sort of)." YMMV :-)

Answer 2

Jarret Hardie already mentioned it: According to Guido van Rossum, there's no better way as of now: As stated in the documentation, join(None) blocks (and that means no signals). The alternative - calling with a huge timeout (join(2**31) or so) and checking isAlive looks great. However, the way Python handles timers is disastrous, as seen when running the python test program with servth.join(100) instead of servth.join():

select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
--- Skipped 15 equal lines ---
select(0, NULL, NULL, NULL, {0, 50000}Killing

I.e., Python wakes up every 50 ms, leading to a single application keeping the CPU from sleeping.

Answer 3

Poll on isAlive before calling join. This polling can be interrupted, of course, and once the thread isn't isAlive, join is immediate.

An alternative would be polling on join with a timeout, checking with isAlive whether the timeout occurred. This can spend less CPU than the previous method.

Answer 4

As far as I understand, a similar question is solved in The Little Book of Semaphores (free download), appendix A part 3…

Answer 5

I know I'm a bit late to the party, but I came to this question hoping for a better answer than joining with a timeout, which I was already doing. In the end I cooked something up that may or may not be a horrible bastardisation of signals, but it involves using signal.pause() instead of Thread.join() and signalling the current process when the thread reaches the end of its execution:

import signal, os, time, sys, threading, random

threadcount = 200

threadlock = threading.Lock()
pid = os.getpid()
sigchld_count = 0

def handle_sigterm(signalnum, frame):
    print "SIGTERM"

def handle_sigchld(signalnum, frame):
    global sigchld_count
    sigchld_count += 1

def faux_join():
    global threadcount, threadlock
    threadlock.acquire()
    threadcount -= 1
    threadlock.release()
    os.kill(pid, signal.SIGCHLD)

def thread_doer():
    time.sleep(2+(2*random.random()))
    faux_join()

if __name__ == '__main__':
    signal.signal(signal.SIGCHLD, handle_sigchld)
    signal.signal(signal.SIGTERM, handle_sigterm)

    print pid
    for i in xrange(0, threadcount):
        t = threading.Thread(target=thread_doer)
        t.start()

    while 1:
        if threadcount == 0: break
        signal.pause()
        print "Signal unpaused, thread count %s" % threadcount

    print "All threads finished"
    print "SIGCHLD handler called %s times" % sigchld_count

If you want to see the SIGTERMs in action, extend the length of the sleep time in thread_doer and issue a kill $pid command from another terminal, where $pid is the pid id printed at the start.

I post this as much in the hope of helping others as being told that this is crazy or has a bug. I'm not sure if the lock on threadcount is still necessary - I put it in there early in my experimentation and thought I should leave it in there in case.