Why doesn't concurrent.futures' submit() immediately returns control to the main program?

Question

I am learning concurrent.futures (moving from threading) and I fail to understand how exactly the submission and waiting work.

Consider the following code, there are two threads started, one of them supposedly immediately finishes while the other indefinitely hangs.

My expectation was that both threads would be stared via submit() and the control immediately given back to the main program.

After that wait() with a timeout would return a tuple of done and not_done threads (the not_done one would be the one forcefully interrupted by the timeout).

import concurrent.futures
import time

def worker(wait_infinite):
    if wait_infinite:
        time.sleep(100)

with concurrent.futures.ThreadPoolExecutor() as executor:
    inf = executor.submit(worker, True)
    not_inf = executor.submit(worker, False)

res = concurrent.futures.wait([inf, not_inf], timeout=2)
print(res)

What happens is that the execution hangs on not_inf = executor.submit(worker, False). Why isn't control given back to the main program?

score 3 · Accepted Answer · edited Jun 20 '20 at 09:12

At the end of the with statement executor the executor.shutdown is called. Waiting on all threads to finish. You should move the wait into the with statement, ~~Then call executor.shutdown(False) to stop waiting on the hanging thread.~~ (edit: you can't correctly cancel a running thread unless cooperatively or if it is in a different process https://stackoverflow.com/a/55351671/7284763)

source

executor.shutdown(wait=True)

Signal the executor that it should free any resources that it is using when the >currently pending futures are done executing. Calls to Executor.submit() and >Executor.map() made after shutdown will raise RuntimeError.

If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed. If wait is False then this method will return immediately and the resources associated with the executor will be freed when all pending futures are done executing. Regardless of the value of wait, the entire Python program will not exit until all pending futures are done executing.

You can avoid having to call this method explicitly if you use the with statement, which will shutdown the Executor (waiting as if Executor.shutdown() were called with wait set to True):

Why doesn't concurrent.futures' submit() immediately returns control to the main program?

1 Answers1