Infinite wait when running a Python ProcessPoolExecutor.map()-method

Question

Infinite wait when running Python ProcessPoolExecutor.map()-method

When n is 10000 to 20000, it works normally, but there is no response until it is forcefully terminated at 100,000.

Other machines with 4 core and 32 GB RAM work, but only this machine does not work.

These are the machine specs :

Any reason?

from concurrent.futures import ProcessPoolExecutor
from tqdm import tqdm

def test_func(x):
    return x+1

if __name__ == '__main__':
    n = 100000
    with ProcessPoolExecutor(4) as p:
        lst = list( tqdm( p.map( test_func,
                                 [i for i in range(n)]
                                 ),
                          total = n,
                          leave = True
                          )
                    )
        # tried the same without a tqdm()-decorated GUI-progress monitor
        #lst = list(      p.map( test_func,
        #                        [i for i in range(n)]
        #                        )
        #                 )

I tried removing tqdm(), increasing ulimit and changing Python version from 3.9 to 3.10, but all did not work.

Your `test_func` isn't a good use case for multiprocessing. But you should actively use `.map`s `chunksize` agrument. E.g. try `p.map(test_func, range(n), chunksize=n // 4)`. See the [docs](https://docs.python.org/3.11/library/concurrent.futures.html#concurrent.futures.Executor.map): _" For very long iterables, using a large value for chunksize can significantly improve performance compared to the default size of 1"_. — Timus, May 26 '23 at 14:05
You souldn't use tqdm in a multiprocessing environment. There are modules for that, see [pqdm](https://stackoverflow.com/questions/37804279/how-can-we-use-tqdm-in-a-parallel-execution-with-joblib) for example. Regarding the number of items, most of the time should be spend in inter-process communication unless `test_func` is expensive to compute 1 item. If `test_func` is cheap, then Python is simply not designed to compute this efficiently (because of IPC for processes and GIL for threads and no other standard solution for parallel computing in CPython yet). — Jérôme Richard, May 27 '23 at 10:59

Infinite wait when running a Python ProcessPoolExecutor.map()-method

0 Answers0