Implementing thread pool for large number of thread classes

Question

There are many examples online for thread pools but almost all deal with the concept of passing functions around and calling those (like here or here). But I have some objects which inherit from threading.Thread and I want only some of them to be running at any time. Here is a minimal working example

class Human (threading.Thread):
    def run (self):
        print ("taking a nap")
        sleep (60*60*8)

human_population = 7000000000

for i in range(human_population):
    human=Human()
    human.start()

Ignoring for the mmoment that 7B objects would crush my computer, I am looking for a very easy way to run only a manageable number of threads at any time, for example, N= 10 threads in a FIFO fashion.

score 2 · Accepted Answer · answered Oct 16 '20 at 16:55

Semaphore is a very fitting tool for these cases. You can use as semaphore with initial size of 10 to control how many threads can be active at most at a time.

A rough implementation over yours (should probably not use a global variable for the semaphore, and handle lifecycles more cleanly):

from threading import Thread, Semaphore
from time import sleep

SEM = Semaphore(10)
 
class Human(Thread):
    def run(self):
        print("taking a nap")
        sleep(5)
        SEM.release() # increments semaphore counter, notifying the job's done

human_population = 20

for i in range(human_population):
    SEM.acquire() # this will block when semaphore has value 0, and will wait until one of the active one finishes and calls release()
    human = Human()
    human.start()

Implementing thread pool for large number of thread classes

1 Answers1