0

Some library has some memory leak, and as a simple solution, I just want to restart my script frequently. My script does some long calculation but I can just save the state and then load it again, so restarting it is not a problem.

For restarting, I was just using os.execv:

os.execv(sys.executable, [sys.executable] + sys.argv)

However, now after a while, I get Too many open files.

Why is that? As far as I know, all file descriptors (fds) should have been closed after exec? I thought we always have the close-on-exec flag set? But maybe not? Or maybe not for all the libraries? Maybe I misunderstood the documentation. Can someone clarify this?

How would I close all fds before the exec? os.closerange? Maybe just:

max_fd = os.sysconf("SC_OPEN_MAX")
os.closerange(3, max_fd)
os.execv(sys.executable, [sys.executable] + sys.argv)

Does that work correctly then?

What are other simple solutions of restarting? I assume fork+exec+exit will just have the same problem then? Or spawn+exit?

Albert
  • 65,406
  • 61
  • 242
  • 386
  • How are you invoking `os.execv`? – Mario Camilleri May 22 '22 at 08:29
  • @MarioCamilleri Just directly? I extended the question. But maybe I did not understand your question. – Albert May 22 '22 at 08:48
  • `exec` does not close any file descriptors. – tripleee May 22 '22 at 08:49
  • `while True: subprocess.run()`? – Aaron May 22 '22 at 08:54
  • @tripleee Not? I thought we always have close-on-exec flag set? – Albert May 22 '22 at 08:57
  • Did any of those questions with same error-message compare or help to yours? [multiprocessing in Python](https://stackoverflow.com/questions/67365870/too-many-open-files-error-with-python-multiprocessing), [subprocess in Python](https://stackoverflow.com/questions/16526783/python-subprocess-too-many-open-files) – hc_dev May 22 '22 at 09:00
  • @hc_dev No I'm not creating a new subprocess here. I just want to restart the app. I ask why `exec` does not work, and how to solve that, or what are other simple solutions. – Albert May 22 '22 at 09:04
  • Which operating system are you using this in? E.g. in a [Python issue 19123](https://bugs.python.org/issue19124) some suggests using `subprocessPopen` or `spawnv` instead `execv`. – hc_dev May 22 '22 at 09:15
  • @hc_dev `spawnv` would spawn a new subprocess which I don't need. Or do you suggest I should use spawnv + exit? – Albert May 22 '22 at 09:16
  • 1
    Well a chance to get a pid or file-object as return, like in `spawnv` or `popen`, would allow to close or `_exit(n)` it. – hc_dev May 22 '22 at 09:36
  • @hc_dev I don't exactly understand your comment. Maybe you can put it into an answer? Or is this actually related to my question (how to restart my script)? – Albert May 23 '22 at 08:26

1 Answers1

1

os.execv does not by itself close any file descriptors. (Think about it: If the exec system call closed file descriptors, there would be no way for the standard fork+exec model to pass on standard input, standard output, and standard error to new processes.) The documentation implicitly mentions this, as it explains that you need to flush any data on any open file handles before calling os.execv.

Like you describe, PEP-446 implemented close-on-exec for most file descriptors created by Python itself. Without access to your code, we can only infer that some of the file descriptors your code creates are outside the scope of the PEP (or created by os.dup2(), where you can control the behavior, but the default is to make the file descriptors inheritable).

Your os.closerange code should work for what you are asking. If you wanted to be more precise, check what files are open in Python has ideas for how to find out which files are open (which could be useful for troubleshooting the root cause, even if you end up using the closerange solution to remediate the immediate problem).

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • Yes, right, it does not make sense that exec closes all file descriptors. But it closes all those with this flag, right? Is this done by Linux, or by Python? In case of fork+exec or spawn, how is the situation then? Does fork closes any fds? Does spawn inherit fds? I guess I should fix some of my own fork+exec code in various projects where I never cared about fds. I guess you always must have an explicit `closerange` then in there, right? – Albert May 23 '22 at 08:24
  • The OS closes any file descriptors on `exec` which were marked as close-on-exec when you created them. Dunno about `spawn`; my impression is that it's only useful on Windows really. – tripleee May 23 '22 at 08:27
  • As for explicit `closerange`, that would only really make sense in long-running processes where you can't otherwise ensure that you don't leak file handles. More of a band-aid, then, than what I would consider recommended practice. – tripleee May 23 '22 at 08:30