0

I am trying to have it so that I can make a number of HTTP get requests to an API at constant time intervals for an indefinite period of time. My idea is to create a Thread pool and limit its size and only once a the Thread Pool has had a thread terminate then it the program will add another thread to it.

Upon some research, I came across this question and this one and the top answers give a solution that works only if you want to make a finite number of GET requests and not an infinite amount.

Quoting the top answer from that question:

import threading
from multiprocessing.pool import ThreadPool

def test_thread(elem):
    return elem ** 2

a = [1,2,3,4,5,6,7,8]
pool = ThreadPool(2) # 2 worker threads
results = []
for x in range(8):
    print x
    results.append(pool.apply_async(test_thread, args=(a[x],)))

results = [result.get() for result in results]
# You can also replace this for loop altogether using pool.map
# and get the same result:
# results = pool.map(test_thread, range(8))
print(results)

I would want to change the for x in range(8): for loop to a while True loop, but doing so would mean that the queue for Thread pool grows to infinity and the computer will produce a memory error.

How can I add threads to a pool only once one has finished without relying on a queue system?

knowledge_seeker
  • 811
  • 1
  • 8
  • 18

1 Answers1

1

Use the concurrent.futures module like this:

futures = set()
# prime the pump
for x in range(8):
    futures.add(pool.submit(test_thread, args=TODO))

# now keep it running with one new task per finished one
while True:
    done, futures = concurrent.futures.wait(futures,
        return_when=concurrent.futures.FIRST_COMPLETED)
    for x in done:
        futures.add(pool.submit(test_thread, args=TODO))
John Zwinck
  • 239,568
  • 38
  • 324
  • 436