1

I'm trying to track down the root cause of this error:

[Errno 24] Too many open files

To better understand how this "open file" leak is happening, I want to be able to check at various points in my program how many open files the program is responsible for.

I read this post, which includes some interesting tips from the resource module, along with this snippet:

import resource
import fcntl

def get_open_fds():
    fds = []
    soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
    for fd in range(0, soft):
        try:
            flags = fcntl.fcntl(fd, fcntl.F_GETFD)
        except IOError:
            continue
        fds.append(fd)
    return fds

The length of fds, I guess, should give you the number of open files your program currently holds.

Is there a more direct way of getting this count? What other strategies should I use to track down these leaks?

For the record, I did also enable ResourceWarning, but that didn't yield any complaints about open files. It did complain about unclosed sockets, but I'm not sure if that can contribute to a "Too many open files" error.

Community
  • 1
  • 1
Nick Chammas
  • 11,843
  • 8
  • 56
  • 115
  • 1
    Sockets have file descriptors accessible via the `.fileno` methods https://docs.python.org/3/library/socket.html#socket.socket.fileno I presume they are counted against the limit. – Terry Jan Reedy Feb 10 '16 at 16:57
  • Following @TerryJanReedy's link, it might be interesting to mention on which OS you are encountering this issue. Depending on how fd's and sockets are handled, this might result in different behaviours. _Just a guess though._ – Dave J Feb 10 '16 at 17:43
  • @DaveJ - I'm running OS X, but for posterity it would interesting to document approaches to tracking leaks that work for multiple platforms. OS X-specific techniques are also welcome, of course. – Nick Chammas Feb 10 '16 at 17:50
  • @TerryJanReedy - Hmm, that might be the root issue then. I'll have to check if open sockets show up in the `fcntl` snippet I posted. Perhaps that's all I need. – Nick Chammas Feb 10 '16 at 18:05
  • Just found out about `psutil`, which has a method specifically for this: http://pythonhosted.org/psutil/#psutil.Process.num_fds – Nick Chammas Feb 13 '16 at 19:53

0 Answers0