The use case I have in mind is as follows: I would like to start a range of jobs with ThreadPoolExecutor and then when a job completes, I would like to add a new job to the queue. I would also like to know when the next job finishes and repeat the procedure above. After I have had the opportunity to observe a predefined number of results, I would like to terminate everything properly. For instance, consider the following code, where the commented out bits in the run_con method show what I would like to achieve.
import numpy as np
import time
from concurrent import futures
MAX_WORKERS = 20
seed = 1234
np.random.seed(seed)
MSG = "Wall time: {:.2f}s"
def expensive_function(x):
time.sleep(x)
return x
def run_con(func, ts):
t0 = time.time()
workers = min(MAX_WORKERS, len(ts))
with futures.ThreadPoolExecutor(workers) as executor:
jobs = []
for t in ts:
jobs.append(executor.submit(func, t))
done = futures.as_completed(jobs)
for future in done:
print("Job complete: ", future.result())
# depending on some condition, add new job to jobs, e.g.
# jobs.append(func, np.random.random())
# update done generator
# if a threshold on total jobs is reached, close every thing down sensibly.
print(MSG.format(time.time()-t0))
ts = np.random.random(10)*5
print("Sleep times: ", ts)
run_con(expensive_function, ts)
Is this achievable with concurrent.futures? If not, what are the alternatives?
