1

When using python's multiprocessing.pool.Pool with more than 63 cores, I get a ValueError:

from multiprocessing.pool import Pool

def f(x):
    return x

if __name__ == '__main__':
    with Pool(70) as pool:
        arr = list(range(70))
        a = pool.map(f, arr)
        print(a)

Output:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\fischsam\Anaconda3\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\fischsam\Anaconda3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\fischsam\Anaconda3\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 72
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69]

The program seems to work fine; the result is just what I expected. Can I ignore the ValueError?

Background: I have googled this issue, and it seems having to do with limitations of python on Windows (_winapi.WaitForMultipleObjects); see e.g. here. The suggested fix is to limit the number of used cores to 63. This is not satisfactory, because I want to 100+ cores on my server. Do I really need to limit the cores? Why? Is there a workaround?

Samufi
  • 2,465
  • 3
  • 19
  • 43

1 Answers1

2

Interestingly, when I went to see if the problem still existed using PrcocessPoolExecutor(max_workers=70) from the concurrent.futures module, I get ValueError: max_workers must be <= 61 and the program terminates immediately before any jobs can be submitted. This strongly suggests a Windows limitation that cannot be circumvented. Your program, however, never actually terminates and just hangs after getting the exception. On my computer with 8 cores, it hangs if I specify any number greater than 60 (not 61 nor 63) workers regardless of whether I use multiprocessing.Pool or concurrent.futures.ProcessPoolExecutor. Find which is the max number of workers on your machine that allows it to terminate normally without generating an exception and stick to it.

from concurrent.futures import ProcessPoolExecutor

def f(x):
    return x

if __name__ == '__main__':
    with ProcessPoolExecutor(max_workers=70) as executor:
        a = list(executor.map(f, range(70)))
        print(a)
Booboo
  • 38,656
  • 3
  • 37
  • 60
  • Thanks for your answer. I understand that the recommendation is to use a number of CPUs that works without causing an exception. My actual question is, however, what could possibly go wrong with using more CPUs and why. Maybe I could live with the drawbacks. Do you have any insights? – Samufi Dec 15 '20 at 14:02
  • As I indicated, when I used more than 60 CPUs, my program never terminated. I consider that to be a major drawback. To each his own, however (chacun à son goût). Also, even though you are specifying more than 60, what makes you think you are actually using the additional cores if Python is not designed to use them? – Booboo Dec 15 '20 at 14:07
  • I can see in the Task Manager how many processes are generated and how much CPU is used. I could live with having to terminate the program manually, as I am not deploying it broadly at this time. Following your comment, however, I have found that the exit block of the context manager is not executed properly. I may continue my search in this direction. – Samufi Dec 15 '20 at 14:31