32

I was using this answer in order to run parallel commands with multiprocessing in Python on a Linux box.

My code did something like:

import multiprocessing
import logging

def cycle(offset):
    # Do stuff

def run():
    for nprocess in process_per_cycle:
        logger.info("Start cycle with %d processes", nprocess)
        offsets = list(range(nprocess))
        pool = multiprocessing.Pool(nprocess)
        pool.map(cycle, offsets)

But I was getting this error: OSError: [Errno 24] Too many open files
So, the code was opening too many file descriptor, i.e.: it was starting too many processes and not terminating them.

I fixed it replacing the last two lines with these lines:

    with multiprocessing.Pool(nprocess) as pool:
        pool.map(cycle, offsets)

But I do not know exactly why those lines fixed it.

What is happening underneath of that with?

cs95
  • 379,657
  • 97
  • 704
  • 746
nephewtom
  • 2,941
  • 3
  • 35
  • 49
  • Here is the [source code](https://github.com/python/cpython/blob/master/Lib/multiprocessing/pool.py#L607-L611) each process is calling `self.terminate()` – salparadise Aug 14 '17 at 00:29
  • 1
    The `with` version is implicitly calling `pool.close()` after `pool.map()` returns. According to the docs, that "Prevents any more tasks from being submitted to the pool. Once all the tasks have been completed the worker processes will exit". This likely causes the open files each task has open to be closed. – martineau Aug 14 '17 at 00:40
  • Am I wrong, or @COLDSPEED did answer the question and now it is erased? I was not able to read it deeply, but I would like to... But now it is gone... Anyone knows why the question is down-voted? – nephewtom Sep 07 '17 at 09:31
  • Hi @cᴏʟᴅsᴘᴇᴇᴅ ... Any hints on this? – nephewtom Oct 17 '17 at 22:57
  • @nephewtom I'd deleted it because I didn't think it was helpful to anyone. I've undeleted it now. – cs95 Oct 17 '17 at 22:58
  • Had the same issue +1 for question! – Geek Mar 24 '18 at 14:49

4 Answers4

32

You're creating new processes inside a loop, and then forgetting to close them once you're done with them. As a result, there comes a point where you have too many open processes. This is a bad idea.

You could fix this by using a context manager which automatically calls pool.terminate, or manually call pool.terminate yourself. Alternatively, why don't you create a pool outside the loop just once, and then send tasks to the processes inside?

pool = multiprocessing.Pool(nprocess) # initialise your pool
for nprocess in process_per_cycle:
    ...       
    pool.map(cycle, offsets) # delegate work inside your loop

pool.close() # shut down the pool

For more information, you could peruse the multiprocessing.Pool documentation.

cs95
  • 379,657
  • 97
  • 704
  • 746
  • Calling manually to `pool.terminate` probably will be the thing to do here. I could not create the pool outside, because I wanted to change it on each iteration. So in each one, the number of spawned processes was increasing. For example, if process_per_cycle was [2, 4, 8], each iteration would spawn 2, 4 and 8 processes. – nephewtom Oct 17 '17 at 23:09
  • @nephewtom Yes, that would be the other option, if you can't use the context manager. – cs95 Oct 17 '17 at 23:10
  • @nephewtom I explained why... because `with` closes the process pool automatically once you're done with it. – cs95 Oct 17 '17 at 23:12
  • Why does the OS treat processes like files in this case? – Sledge Dec 18 '19 at 18:06
  • @Sledge When a process is created it must be explicitly destroyed unless it's a daemon. – cs95 Dec 18 '19 at 18:13
3

I was already terminating and closing pool but there was a limit on number of file descriptors and I changed my ulimit to 4096 from 1024 and it worked. Following is the procedure:

Check:

ulimit -n

I updated it to 4096 and it worked.

ulimit -n 4096
devil in the detail
  • 2,905
  • 17
  • 15
2

It is context manger. Using with ensures that you are opening and closing files properly. To understand this in detail, I'd recommend this article https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/

pritesh
  • 193
  • 1
  • 4
  • 18
  • It's obvious it is a context manager. But why does its use here cause this to work? – cs95 Aug 14 '17 at 00:26
  • You can have a look at [this](https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/) explanation. – rpanai Jun 20 '18 at 17:21
1

This can happen when you use numpy.load too, make sure close those files too, or avoid using it and use pickle or torch.save torch.load etc.

JAZ
  • 11
  • 3