Is there a way to manually get out of a multithreaded operation? Here is what I'm trying to do:
pool = Pool(num_parallel_workers)
pool.map(update_completions_in_parallel, list_of_hits)
# press ctrl+c in middle of operation to 'exit' and return to CLI prompt
pool.close()
pool.join()
Why not just use an iterated or asynchronous map? These are easy to interrupt or kill or whatever as the map is not blocking.
See a good usage example here: Python multiprocessing - tracking the process of pool.map operation
I used pathos for my answer linked above, but there are also imap and map_async available from plain old multiprocessing.
>>> import multiprocessing as mp
>>> p = mp.Pool()
>>> p.imap
<bound method Pool.imap of <multiprocessing.pool.Pool object at 0x109592c50>>
>>> p.map_async
<bound method Pool.map_async of <multiprocessing.pool.Pool object at 0x109592c50>>
However, if you are looking to terminate the processes/threads that were launched already by doing a close then join… and then save cycles… what you typed (i.e. close then join) is not going to work. You have to use terminate to do that… and terminating a pool is not recommended if you plan on touching the pool or result object afterward. Better to use an iterated/asynchronous map, then when see you have what you want -- just ignore the rest and move on.
