5

I have a python script that runs multiprocessing.Pool to process a lot of files separately. I usually have a cpu limit of 8. My problem is after running a while I always get "IOError: [Errno 24] Too many open files". Each child process opens a few files for reading only with file.open(). These file handlers are than passed to multiple functions to retrieve data. At the end of the each child process these files are closed with file.close(). I tried the with statement as well but did not fix the issue. Does any one have any idea whats wrong. I googled around but failed to find any answers. I am closing the files and the functions are returning properly so what keep the file handlers around.

My settings are Mac 10.5 with python 2.6

Thanks

Ogan

    from custom import func1, func2
    # func1 and func2 only seek, read and return values form the file
    # however, they do not close the file
    import multiprocessing
    def Worker(*args):
        f1 = open("db1.txt")
        f2 = open("db2.txt")
        for each in args[1]:
            # do many stuff
            X = func1(f1)
            Y = func2(f2)

        f1.close()
        f2.close()
        return

    Data = {1:[2], 2:[3]}  
    JobP= multiprocessing.Pool(8) 
    jobP.map_async(Worker, Data.items()) 
    jobP.close()
    jobP.join()
Ogan
  • 51
  • 1
  • 3
  • 1
    Show us your code so we can see when the files get closed, etc. – agf Jul 22 '11 at 13:48
  • @agf. Thanks for the help. It is very long to put it here the basic idea. – Ogan Jul 22 '11 at 14:38
  • 1
    If it's too long to post you need to cut it down to the minimum code to trigger the problem, then post that. In doing so you might even find the problem yourself. – agf Jul 22 '11 at 14:42
  • it looks like you have just the two files ... and are writing to them w/ multiple processes or just reading? If you have multiple processes writing to the same file w/o protection around the files you can definitely get an ioerror – pyInTheSky Jul 22 '11 at 19:54

2 Answers2

0

Its likely you are being limited by the operating system's open file limit. See How do I change the number of open files limit in Linux? for more information. I personally prefer to change the /etc/security/limits.conf settings.

Community
  • 1
  • 1
BenH
  • 894
  • 8
  • 7
0

To change the number open files limit in Yosemite (OS X 10.10):

sudo launchctl limit maxfiles [number-of-files] unlimited
cookedapple
  • 71
  • 7