1

I am running a python script which uses scipy.optimize.differential_evolution to find optimum parameters for given data samples. I am processing my samples sequentially. The script is running fine, although when I wanted to make use of the parallel computing options implemented in the package, by calling it via:

res = scipy.optimize.differential_evolution( min_fun,
                                             bounds   =     bnds,
                                             updating =    'deferred',
                                             workers  =  120
                                             )

after evaluating res for a few times, it throws an error

File "[...]/miniconda3/envs/remote_fit_environment/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

If I allocate less CPUs, e.g. workers=20 it takes longer, as in more times calling differential_evolution() , until the error occurs.

I am aware that I could raise the limit for open files (1024 by default on that machine), but this seems strange to me. Since the Error origins in the multiprocessing module and is traced back to differential_evolution, I am wondering whether something with the parallelisation implementation within scipy.optimize.differential_evolution might be wrong or "not clean" (although it is much more likely that I am missing something as I am completely new to this whole parallel/multiprocessing thing)?

Any similar experiences or ideas how to solve this?

user3666197
  • 1
  • 6
  • 50
  • 92
chrizz
  • 43
  • 5
  • 120 workers does not seem reasonable to me (at least on PC and most servers). Even if it would be faster, it should take far more system resources than probably needed (120 stack to allocates and 120 processes to create, schedule and wait). Do you have a processor with a lot of core (like 64)? The optimal number should be the number of hardware threads (generally twice the number of cores). You should also care about the scheduling of processes by mapping them on hardware threads. – Jérôme Richard May 13 '22 at 14:59
  • @JérômeRichard It's a server where 160 cores are available to me... 120 was just a completely arbitarly number to check out this parallel processing option. As mentioned above, the error also occurs when reducing the number of cores. What would you suggest i should do here? – chrizz May 16 '22 at 08:33
  • Ha Ok, I though you were on a mainstream machine. You can increase the file descriptor limit so. (Are you sure this is cores and not threads btw? 160 core-machines are very rare -- certainly 8 sockets of 20 cores each since there are no 40, 80 or 160 cores mainstream processors yet AFAIK). – Jérôme Richard May 16 '22 at 08:41
  • @JérômeRichard Yeah you are right! i just checked it again: it's 160 logical cores (physical: 4 sockets x 20 cores each). So is it "save" to just increase the file limit? What throws me of is that multiple fits are completed successfully via ```scipy.optimize.differential_evolution()``` before it crashes eventually. That makes me think that within this module suprocesses are not terminated correctly or sth!? – chrizz May 16 '22 at 09:17

1 Answers1

0

The root cause is in O/S not having enough "file-descriptors" available for so many process-to-process IPC channels ( as Python IPC as in joblib.parallel() or multiprocessing modules use os.pipe-s for communicating parameters between the "main"-Python-Interpreter process and the (safe)-spawn()-ed / (unsafe, as documented elsewhere why)-fork()-ed sub-ordinate processes, which SciPy re-wraps and re-uses under the calling-signature workers = 120 parameter.

Even if you "increase" the configured amount of O/S filedescriptors ( which intelligent operating systems permit ) for having more os.pipe-instances to work further, the whole intent is not reasonable, as the end-to-end performance will annihilate, as 128-worker processes will have unresolvably self-devastating bottlenecks on CPU-to-RAM physical-RAM-I/O channels on such "over-booked" CPU-to-RAM I/O-traffic density.

Asking more you easily receive less ( or nothing at all )

Details matter.

user3666197
  • 1
  • 6
  • 50
  • 92
  • Thanks for your answer, unfortunately I don't really know what make of it. I am getting your point that parts of the system infrastructure other than the number of cores are limiting computing speed. What would be a reasonable number of ```workers```to assign to the optimization task (i have 160 physical cores available to me). – chrizz May 16 '22 at 08:40