How to limit the number of Threads

Question

There are 101 things stored in THINGS variable. The code declares 101 threads and executes them all at once instantly all at the same time.

I wonder if we can limit the number of the active threads to just 12.

At first only 12 threads should pick their 12 THINGS to process. The rest of the threads should be waiting for the first 12 to finish their jobs. When the first 12 threads are all done then the next 12 threads would pick up the next 12 THINGS to process. And so one.

Would it be possible?

import Queue
import threading, time

class MyThread(threading.Thread):
    def __init__(self, theQueue=None):
        threading.Thread.__init__(self)        
        self.theQueue=theQueue

    def run(self):
        thing=self.theQueue.get()
        self.process(thing) 
        self.theQueue.task_done()

    def process(self, thing):
        time.sleep(1)
        print 'processing %s'%thing.name

queue=Queue.Queue()
THINGS = ['Thing%02d'%i for i in range(101)]

THREADS=[]
for thing in THINGS:
    thread=MyThread(theQueue=queue)
    thread.name = thing
    THREADS.append(thread)
    thread.start() 

for thread in THREADS:       
    queue.put(thread)

Possible duplicate of [The right way to limit maximum number of threads running at once?](http://stackoverflow.com/questions/19369724/the-right-way-to-limit-maximum-number-of-threads-running-at-once) — Morishiri, Apr 30 '16 at 07:00
Looks like a pool.. https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers — Olegp, Apr 30 '16 at 07:02
I wonder if we can limit the number of the active threads to just 12' sure. Make 12 threads. Don't make any more. Don't let any of them terminate. — Martin James, Apr 30 '16 at 09:18
Due to the fact that Python uses a [Global Interpreter Lock](https://wiki.python.org/moin/GlobalInterpreterLock), multithreading doesn't make sense for parallel computations—you need to use multiprocessing for that (as suggested by Olegp). Threading in Python is really only useful for concurrent applications, especially where several threads might block at the same time. Examples might be GUI applications, or multiple I/O requests. (There are a couple implementations with no GIL, but it doesn't look like you're using one of those.) — DaoWen, Apr 30 '16 at 15:23

alphanumeric · Accepted Answer · 2016-04-30T16:13:10.193

The working solution is posted below. The basic idea is that we declare only as many Threads instances as there are available CPUs. Then we proceed by adding the "tasks" (or "things" here) to the Queue. As soon as the task is added to the queue it is being immediately picked up by one of the Thread instances we declared in the previous step.

Important: In order for this mechanism to work the MyThread.run() method should be running inside of the while loop. Otherwise MyThread instance will be terminated as soon as it completes the very first task. The while loop will exit itself after no tasks in the Queue are left. That is the end of story.

import Queue
import threading, time

class MyThread(threading.Thread):
    def __init__(self, theQueue=None):
        threading.Thread.__init__(self)        
        self.theQueue=theQueue

    def run(self):
        while True:
            thing=self.theQueue.get()
            self.process(thing) 
            self.theQueue.task_done()

    def process(self, thing):
        time.sleep(1)
        print 'processing %s'%thing

queue=Queue.Queue()
THINGS = ['Thing%02d'%i for i in range(101)]
AVAILABLE_CPUS=3

for OneOf in range(AVAILABLE_CPUS):
    thread=MyThread(theQueue=queue)
    thread.start() # thread started. But since there are no tasks in Queue yet it is just waiting.

for thing in THINGS:       
    queue.put(thing) # as soon as task in added here one of available Threads picks it up

Brilliant answer! Very helpful. The only thing that didn't work for me, the code never terminated so I used `while not theQueue.empty():` instead of `while True`. To make this work, I ran the `for` loop with `THIINGS` before running the `for` loop with `AVAILABLE_CPUS` otherwise the Thread would terminate immediately because `myQueue` would be empty. — M3RS, Sep 06 '17 at 08:24

How to limit the number of Threads

1 Answers1

Linked