Danger to having long lasting (non-deamon) threads in a Django/Gunicorn app?

Question

I generally don't need to explicitly use threads in my Django application level programming (i.e. views). But I've noticed a library that looks interesting which handles server side analytics by via threading.

During a Django view, you would use their Python client to batch HTTP POSTs to their web service in a separate (non-daemon) thread. Normally, I would go with RabbitMQ for something like this, instead of threads but they wanted to lower the startup costs for the library.

My question is, are there any downsides to this approach? Threads have some additional memory footprint, but I'm not too worried about that. It obviously depends on the number of requests/threads started.

Is the fact that the threads are not daemons and potentially long running an issue? I assume that the Gunicorn process is the main thread of execution and it runs in an infinite loop, so it generally doesn't matter if it has to wait on the non-daemon threads to exit. Is that correct?

Kind of an open question but the main point is understanding the impact of non-daemon threads in Django/Gunicorn apps.

Answer 1

Gunicorn uses a pre-fork worker model. The Master process spawns and manages Worker processes. For non-Tornado uses, there are two kinds of Workers: Sync (default) and Async.

In normal operations, these Workers run in a loop until the Master either tells them to graceful shutdown or kills them. Workers will periodically issue a heartbeat to the Master to indicate that they are still alive and working. If a heartbeat timeout occurs, then the Master will kill the Worker and restart it.

Therefore, daemon and non-daemon threads that do not interfere with the Worker's main loop should have no impact. If the thread does interfere with the Worker's main loop, such as a scenario where the thread is performing work and will provide results to the HTTP Response, then consider using an Async Worker. Async Workers allow for the TCP connection to remain alive for a long time while still allowing the Worker to issue heartbeats to the Master.