In Python threading, how I can I track a thread's completion?

Question

I've a python program that spawns a number of threads. These threads last anywhere between 2 seconds to 30 seconds. In the main thread I want to track whenever each thread completes and print a message. If I just sequentially .join() all threads and the first thread lasts 30 seconds and others complete much sooner, I wouldn't be able to print a message sooner -- all messages will be printed after 30 seconds.

Basically I want to block until any thread completes. As soon as a thread completes, print a message about it and go back to blocking if any other threads are still alive. If all threads are done then exit program.

One way I could think of is to have a queue that is passed to all the threads and block on queue.get(). Whenever a message is received from the queue, print it, check if any other threads are alive using threading.active_count() and if so, go back to blocking on queue.get(). This would work but here all the threads need to follow the discipline of sending a message to the queue before terminating.

I'm wonder if this is the conventional way of achieving this behavior or are there any other / better ways ?

Answer 1

Here's a variation on @detly's answer that lets you specify the messages from your main thread, instead of printing them from your target functions. This creates a wrapper function which calls your target and then prints a message before terminating. You could modify this to perform any kind of standard cleanup after each thread completes.

#!/usr/bin/python

import threading
import time

def target1():
    time.sleep(0.1)
    print "target1 running"
    time.sleep(4)

def target2():
    time.sleep(0.1)
    print "target2 running"
    time.sleep(2)

def launch_thread_with_message(target, message, args=[], kwargs={}):
    def target_with_msg(*args, **kwargs):
        target(*args, **kwargs)
        print message
    thread = threading.Thread(target=target_with_msg, args=args, kwargs=kwargs)
    thread.start()
    return thread

if __name__ == '__main__':
    thread1 = launch_thread_with_message(target1, "finished target1")
    thread2 = launch_thread_with_message(target2, "finished target2")

    print "main: launched all threads"

    thread1.join()
    thread2.join()

    print "main: finished all threads"

Answer 2

The thread needs to be checked using the Thread.is_alive() call.

Answer 3

Why not just have the threads themselves print a completion message, or call some other completion callback when done?

You can the just join these threads from your main program, so you'll see a bunch of completion messages and your program will terminate when they're all done, as required.

Here's a quick and simple demonstration:

#!/usr/bin/python

import threading
import time

def really_simple_callback(message):
    """
    This is a really simple callback. `sys.stdout` already has a lock built-in,
    so this is fine to do.
    """    
    print message

def threaded_target(sleeptime, callback):
    """
    Target for the threads: sleep and call back with completion message.
    """
    time.sleep(sleeptime)
    callback("%s completed!" % threading.current_thread())

if __name__ == '__main__':
    # Keep track of the threads we create
    threads = []

    # callback_when_done is effectively a function
    callback_when_done = really_simple_callback

    for idx in xrange(0, 10):
        threads.append(
            threading.Thread(
                target=threaded_target,
                name="Thread #%d" % idx,
                args=(10 - idx, callback_when_done)
            )
        )

    [t.start() for t in threads]
    [t.join() for t in threads]

    # Note that thread #0 runs for the longest, but we'll see its message first!

Answer 4

What I would suggest is loop like this

while len(threadSet) > 0:
    time.sleep(1)
    for thread in theadSet:
        if not thread.isAlive()
            print "Thread "+thread.getName()+" terminated"
            threadSet.remove(thread)

There is a 1 second sleep, so there will be a slight delay between the thread termination and the message being printed. If you can live with this delay, then I think this is a simpler solution than the one you proposed in your question.

Answer 5

You can let the threads push their results into a threading.Queue. Have another thread wait on this queue and print the message as soon as a new item appears.

Answer 6

I'm not sure I see the problem with using: threading.activeCount()

to track the number of threads that are still active?

Even if you don't know how many threads you're going to launch before starting it seems pretty easy to track. I usually generate thread collections via list comprehension then a simple comparison using activeCount to the list size can tell you how many have finished.

See here: http://docs.python.org/library/threading.html

Alternately, once you have your thread objects you can just use the .isAlive method within the thread objects to check.

I just checked by throwing this into a multithread program I have and it looks fine:

for thread in threadlist:
        print(thread.isAlive())

Gives me a list of True/False as the threads turn on and off. So you should be able to do that and check for anything False in order to see if any thread is finished.

Answer 7

I use a slightly different technique because of the nature of the threads I used in my application. To illustrate, this is a fragment of a test-strap program I wrote to scaffold a barrier class for my threading class:

   while threads:
        finished = set(threads) - set(threading.enumerate())
        while finished:
            ttt = finished.pop()
            threads.remove(ttt)
        time.sleep(0.5)

Why do I do it this way? In my production code, I have a time limit, so the first line actually reads "while threads and time.time() < cutoff_time". If I reach the cut-off, I then have code to tell the threads to shut down.