What happens to threads in python when there is no .join()?

Question

Suppose, we have a multi-thread Python code which looks like this:

import threading
import time

def short_task():
    print 'Hey!'

for x in range(10000):
    t = threading.Thread(target=short_task)
    t.daemon = True  # All non-daemon threads will be ".join()"'ed when main thread dies, so we mark this one as daemon
    t.start()

time.sleep(100)

Are there any side-effects from using similar approach in long-running applications (e.g. Django+uwsgi)? Like no garbage collection, extra memory consumption, etc?

What I am trying to do is to do some costly logging (urlopen() to external API url) without blocking the main thread. Spawning infinite new threads with no .join() looks like best possible approach here, but maybe I am wrong?

Answer 1

Not a 100% confident answer, but since nobody else has weighed in...

I can't find any place in the Python documentation that says you must join threads. Python's threading model looks Java-like to me: In Java t.join() means "wait for t to die," but it does not mean anything else. In particular, t.join() does not do anything to thread t.

I'm not an expert, but it looks like the same is true in Python.

---

Are there any side-effects...Like...extra memory consumption

Every Python thread must have its own, fixed-size call stack, and the threading module documentation says that the minimum size of a stack is 32K bytes. If you create ten thousand of those, like in your code snippet, and if they all manage to exist at the same time, then just the stacks alone are going to occupy 320 megabytes of real memory.

It's unusual to find a good reason for a program to have that many simultaneous threads.

If you're expecting those threads to die so quickly that there's never more than a few of them living at the same time, then you probably could improve the performance of your program by using a thread pool. A thread pool is an object that manages a small number of worker threads and a blocking queue of tasks (i.e., functional objects). Each worker sits in a loop, picking tasks from the queue and performing them.

A program that uses a thread pool effectively re-uses its worker threads instead of continually letting threads die and creating new ones to replace them.