Python threading in a loop but with max threads

Question

Right now I have a for loop that loops through a list, usually this list is 100-500 items long. In the for loop, a new thread is opened per item. So right now my code looks like this:

    threads = []
    for item in items:
        t = threading.Thread(target=myfunction, args=(item,))
        threads.append(t)
        t.start()

But I don't want to start a new thread per, seeing it only takes a few seconds MAX per thread to execute myfunction. I want to do my loop still, calling upon myfunction with each item in argument. But to close the thread once it's done, and allow another one to take over. The max number of threads I want to open is no less than 3, no more than 20. Although if it's easier, that range can vary. I just don't want to open up a new thread each item in the loop.

For those that are curious, and if it matters. myfunction is a function that I defined that uses urllib to send a post request to a site.

I'm new to python, but I'm not new to coding all together. Sorry for the noob question.

`threading.activeCount()` can help you decide if to spawn a thread or execute `myfunction` there and then, passing it the current `item` — Pynchia, Jun 12 '15 at 22:04
Okay. So would my code look like if threading.activeCount()>20 than just execute myfunction, else start thread? — user1687621, Jun 12 '15 at 22:15
yep, I have added it as an answer, I took some time to test it — Pynchia, Jun 12 '15 at 22:50

score 7 · Answer 1 · edited May 23 '17 at 12:02

7

I think you are looking for a thread pool to solve your problem.

The answers to this question detail some possible solutions.

One of the simplest (assuming python3 or the backport in pypi) is:

from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=10)
futures = []
for item in items:
    a = executor.submit(myfunction, item)
    futures.append(a)

This will execute myfunction for all items using 10 threads. You can later wait for completion of the calls using the futures list.

edited May 23 '17 at 12:02

Community

1
1

answered Jun 12 '15 at 22:32

textshell

1,746
14
21

This needs to be marked as the answer - it's simple and worked like a charm for me. – Cody Jan 25 '21 at 17:57

score 1 · Answer 2 · answered Jun 12 '15 at 22:12

i believe your issue lies with the missing functions. it could be a number of issues, I Recommend you visit pythons home page: https://goo.gl/iAZuNX

#!/usr/bin/python

import thread
import time

# Define a function for the thread
def print_time( threadName, delay):
   count = 0
   while count < 5:
      time.sleep(delay)
      count += 1
      print "%s: %s" % ( threadName, time.ctime(time.time()) )

# Create two threads as follows
try:
   thread.start_new_thread( print_time, ("Thread-1", 2, ) )
   thread.start_new_thread( print_time, ("Thread-2", 4, ) )
except:
   print "Error: unable to start thread"

Pynchia · Answer 3 · 2015-06-12T22:48:50.097

modify your code slightly to include the check on the number of active threads at any given time:

threads = []
consumed_by_threads = 0
consumed_by_main = 0
for item in items:
    at = threading.activeCount()
    if at <= 20:
        t = threading.Thread(target=myfunction, args=(item,))
        threads.append(t)
        consumed_by_threads += 1
        t.start()
    else:
        print "active threads:", at
        consumed_by_main += 1
        myfunction(item)

print "consumed_by_threads: ", consumed_by_threads
print "consumed_by_main: ", consumed_by_main

# here the rest of your code, thread join, etc

Note: I am only checking for a max number of threads. BTW: it should be 21, since the main thread is included in the count (see here and follow the link to enumerate)

Nota Bene: as usual, double check the benefit of multithreading on your specific application, depending on which python implementation you use and whether the threads are cpu-bound or I/O bound.

Python threading in a loop but with max threads

3 Answers3