0

I have a python script that needs to load lot's of data by calling lot's of external commands. After a couple hours this always crashes with this:

....
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
File "/usr/lib/python2.6/subprocess.py", line 621, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1037, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Not enough space
abort: Not enough space

... even though the machine does have much more memory available than the script is actually using.

The root cause seems to be that every fork() actually requires twice as much memory than the parent process that is immediately released by calling of exec() (see: http://www.oracle.com/technetwork/server-storage/solaris10/subprocess-136439.html) ... in my case the above is even worse because I'm loading the data in multiple threads.

So do you see any creative way how to workaround this issue?

Cœur
  • 37,241
  • 25
  • 195
  • 267
pez
  • 11
  • 2

1 Answers1

-1

Do you need to just launch an external command or fork the one you're running?

If you just need to run another script try:

subprocess.call()

i.e.

subprocess.call(["ls", "-l"])

The os.fork copies your current environment which might be unnecessary depending on your usage.

Don't forget to import the subprocess module before using it.

More info:

Community
  • 1
  • 1
droghio
  • 90
  • 7
  • The problem with using subprocess.call() is that it is in fact using os.fork() (followed by os.execvp()), and that in turn calls the syscall fork(). The syscall fork() on Solaris (and other U*IX systems that doesn't have memmory overcommitment enabled) in fact "duplicates" the process including all memmory regions the process allocated (while the following syscall execvp() actually releases most of it in a while). So if you have a process with more than 1/2 of systems memmory allocated the syscall fork() fails, no matter how small the subprocess is :-( – pez Jul 02 '14 at 12:32