5

A python script executes an IO-bound function a lot of times (order of magnitude: anything between 5000 and 75000). This is still pretty performant by using

def _iterator(): ...  # yields 5000-75000 different names
def _thread_function(name): ...

with concurrent.futures.ThreadPoolExecutor(max_workers=11) as executor:
    executor.map(_thread_function, _iterator(), timeout=44)

If a user presses CTRL-C, it just messes up a single thread. I want it to stop launching new threads; and finish the current ongoing threads or kill them instantly, whatever.

How can I do that?

parvus
  • 5,706
  • 6
  • 36
  • 62

1 Answers1

2

Exception handling in concurrent.futures.Executor.map might answer your question.

In essence, from the documentation of concurrent.futures.Executor.map

If a func call raises an exception, then that exception will be raised when its value is retrieved from the iterator.

As you are never retrieving the values from map(), the exception is never raised in your main thread.

Furthermore, from PEP 255

If an unhandled exception-- including, but not limited to, StopIteration --is raised by, or passes through, a generator function, then the exception is passed on to the caller in the usual way, and subsequent attempts to resume the generator function raise StopIteration. In other words, an unhandled exception terminates a generator's useful life.

Hence if you change your code to (notice the for loop):

def _iterator(): ...  # yields 5000-75000 different names
def _thread_function(name): ...

with concurrent.futures.ThreadPoolExecutor(max_workers=11) as executor:
    for _ in executor.map(_thread_function, _iterator(), timeout=44):
        pass

The InterruptedError will be raised in the main thread, and by passing through the generator (executor.map(_thread_function, _iterator(), timeout=44)) it will terminate it.

Boyan Hristov
  • 1,067
  • 3
  • 15
  • 41
  • When I do that simple change, I get hit with a TimeoutError after some time. I don't understand how this can affect timing. – parvus Oct 31 '20 at 09:12
  • On the plus side, I can catch an exception now, and call executor.shutdown. It takes a while before the exception gets handled, which is annoying, since I can't immediately provide feedback to the user: a print statement takes ~20 seconds to show. With this, the script exits after a minute or so, which is definitely a massive improvement. If only that timing wasn't changed so drastically. – parvus Oct 31 '20 at 09:16
  • 1
    It could be that previously the exception was still raised but you never received it in the main thread, as the result of that job was never retrieved from the iterator (see quote of map documentation). – Boyan Hristov Nov 01 '20 at 15:54
  • Will it work in windows 10? – James Jun 07 '22 at 10:31
  • I don't see why it shouldn't. – Boyan Hristov Jun 07 '22 at 11:53