1

I create a process pool with four processes and use imap_unordered to compute things in parallel and get intermediate results.

The code looks like this:

import multiprocessing


def f(x):
    print(f"Start {x} on {multiprocessing.current_process().name}")
    # do stuff
    print(f"Finish {x} on {multiprocessing.current_process().name}")


def main():
    pool = multiprocessing.Pool(processes=4)
    iterable_in = [i for i in range(1000)] # input iterable
    for result in pool.imap_unordered(f, iterable_in, chunksize=5):
        # do stuff with result
        pass


if __name__ == "__main__":
    main()

The output looks as follows:

...
Finish 71 on ForkPoolWorker-4
Start 72 on ForkPoolWorker-4
Finish 76 on ForkPoolWorker-3
Start 77 on ForkPoolWorker-3
Finish 63 on ForkPoolWorker-1
Start 64 on ForkPoolWorker-1
Start 85 on ForkPoolWorker-5
Finish 73 on ForkPoolWorker-4
Start 74 on ForkPoolWorker-4
Finish 68 on ForkPoolWorker-2
Start 69 on ForkPoolWorker-2
Finish 85 on ForkPoolWorker-5
Start 86 on ForkPoolWorker-5

For the first iterations it looks fine, but at some point a fifth worker seems to be spawned continuing with a new chunk. From that point on, only workers 1, 2, 4 and 5 seems to proceed and worker 3 never finishes its current chunk. htop shows a continuous load average of 4.00 and only four child processes.

Any ideas what could happened here?

Yannic
  • 698
  • 8
  • 22
  • If there's an error in one of the workers and it terminates, there might be a new worker spawned. – Dschoni Feb 03 '21 at 12:09
  • Also, could you please post an minimal, reproducible example? Your example throws errors as is. – Dschoni Feb 03 '21 at 12:11
  • @Dschoni, Sorry, corrected the example. But is there any way to get more information why the worker gets terminated? Any exception raised inside the worker should just produce an error output and continue with the next chunk sample without killing the worker, or not? – Yannic Feb 03 '21 at 12:22
  • Yes. I asked a question some years ago for this exact same question. https://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing – Dschoni Feb 03 '21 at 12:25
  • By the way, for me, the code prints only 4 distinct numbers in workers, so it should be working as expected. – Dschoni Feb 03 '21 at 12:37
  • Since this is the first and only search result for this issue, my problem was that numpy was starting more processes inside my multiprocessed function: https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy – Moo Jan 01 '23 at 17:42

0 Answers0