0

I'd like to have something like Executors.newFixedThreadPool(3) which is in Java.

That is to say, I want my python program to fork at most 3 multi thread at a time, if the number of multi thread is 3 for now, wait until the number is less than 3.

Is there anything like that in python? Thanks a lot!

Judking
  • 6,111
  • 11
  • 55
  • 84
  • Something like [this](http://docs.python.org/2.6/library/multiprocessing.html#using-a-pool-of-workers)? – RedBaron Nov 18 '13 at 05:27
  • If I didn't get that wrong, it's for multiprocessing? What I want is for multithreading. Anyway, thanks for a silver lining :) @RedBaron – Judking Nov 18 '13 at 05:31
  • 1
    Yes it is. You can see [this question](http://stackoverflow.com/questions/3033952/python-thread-pool-similar-to-the-multiprocessing-pool) for threads. Aside, IMO GIL really inhibits threading in Python and if possible you should consider multiprocessing – RedBaron Nov 18 '13 at 05:33

2 Answers2

3

I think the what you need is a semaphore object:

threadLimiter = threading.BoundedSemaphore(maximumNumberOfThreads)

class YourThread(threading.Thread):

    def run(self):
        threadLimiter.acquire()
        try:
            <your code here>
        finally:
            threadLimiter.release()

When you start threads more than maximum number of threads, the exceeding threads will wait in threadLimiter.acquire() till the end of some threads.

scriptmonster
  • 2,741
  • 21
  • 29
1

If you're running python 3.2+, you have access to concurrent.futures, a high-level threading module which conveniently refers to its interface as Executor. It comes in both ThreadPoolExecutor and ProcessPoolExecutor flavors.

#adapted example from docs
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    future_results = [executor.submit(my_func, *args) for _ in range(100000)]
    for result in concurrent.futures.as_completed(future_results):
        #do something with each result

More fleshed-out example of retrieving the results of many urllib.request calls can be found in the docs. It should be noted that that is what you should be using many threads for - tasks that are I/O bound. Tasks that are processor-bound will not benefit from multithreading due to the GIL - for that, use ProcessPoolExecutor (which leverages multiprocessing).

roippi
  • 25,533
  • 4
  • 48
  • 73
  • @Judking bummer. I'll leave the answer up in case it helps someone else. Good luck working with those `threading` primitives :) – roippi Nov 18 '13 at 06:40