subprocess.check_output without high memory usage

Question

In my current project I have a webserver that calls Linux commands to get information that is then displayed on the website. The problem I have with that is that the webserver runs on a tiny embedded device (it is basically a configuration tool for the device) that has only 256 MB of RAM. The webserver itself does take more than half of the free RAM that I have on that device.

Now when I try to use subprocess.check_output() to call a command the fork shortly doubles the RAM usage (because it clones the parent process or something, as far as I understand) and thus crashes the whole thing with an "Out of Memory", though the called process is quite tiny.

Since the device uses pretty cheap flash chips that have proven to fail if overused I don't want to use any swap solutions or other solutions that are based on increasing the virtual memory.

What I tried to do so far is to Popen a sh session at the start of the program when it is still low on memory usage and then write the commands to that sh session and read the output. This kinda works, but it is quite unstable, since a wrong "exit" or something therelike can crash the whole thing.

Is there any solution similar to subprocess.check_output() that doesn't double my memory usage?

I don't think that would help much, since the problem is not so much about the webserver itself, but rather about calling other programs from within python. I have the same problem on a command line utility that I made for the same purpose (configuring that device), but to a lesser extent, since the CLI-tool is much smaller. Also I can't share the source, since I build that program as part of my job and they wouldn't let me share the full source. — Dakkaron, Jun 15 '15 at 14:27
Right; *fair enough*. There are two ways I can *think* of to solve this. 1) Reduce the memory footprint of your webserver/app 2) Run another daemon separate to your webserver/app that can be communicated with over a UNIX Socket/Pipe. — James Mills, Jun 15 '15 at 15:04
The second solution sounds good. I will take a look at that, thanks a lot! — Dakkaron, Jun 15 '15 at 15:33
The issue is due to `os.fork()` and the standard solution is a fork-server (a dedicated process that spawns other processes -- `multiprocessing`-based instead of `sh`-based as in your question). Related: [Python subprocess.Popen “OSError: [Errno 12\] Cannot allocate memory”](http://stackoverflow.com/q/1367373/4279) — jfs, Jun 16 '15 at 00:50
@J.F.Sebastian: Thanks a lot for that hint, I will try that! Would you like to put that into an answer so I can upvote that? — Dakkaron, Jun 16 '15 at 08:41
@Dakkaron: it is just an idea. Try it to see whether `multiprocessing.Process` is too heavy for 256MB RAM (you need at least two processes the main script and the fork-server) and [post it as your own answer](http://stackoverflow.com/help/self-answer) — jfs, Jun 16 '15 at 16:36
Thanks a lot, it actually worked. It is kinda heavy (the fork-server takes about 19 MB of RAM, but I might be able to reduce that a little), but it's a lot better than anything I had before. I'll post it as an answer, just thought you might want the rep for it. Thanks a lot! — Dakkaron, Jun 17 '15 at 07:26

score 7 · Accepted Answer · edited Aug 15 '16 at 11:11

7

So with the help of J.F. Sebastian I figured it out.

This is the code I used in the end:

from multiprocessing import Process, Queue
from subprocess import check_output, CalledProcessError

def cmdloop(inQueue,outQueue):
    while True:
        command = inQueue.get()
        try:
            result = check_output(command,shell=True)
        except CalledProcessError as e:
            result = e

        outQueue.put(result)

inQueue = Queue()
outQueue = Queue()
cmdHostProcess = Process(target=cmdloop, args=(inQueue,outQueue,))
cmdHostProcess.start()

def callCommand(command):
    inQueue.put(command)
    return outQueue.get()

def killCmdHostProcess():
    cmdHostProcess.terminate()

In Python 3.4+ I could have used multiprocessing.set_start_method('forkserver'), but since this runs on Python 2.7 this is sadly not available.

Still this reduces my memory usage by a long shot and removes the problem in a clean way. Thanks a lot for the help!

edited Aug 15 '16 at 11:11

Mark Amery

143,130
81
406
459

answered Jun 17 '15 at 10:51

Dakkaron

5,930
2
36
51

1

This is a nice solution to the problem, but the fact that Python makes this necessary in the first place is awful. – Mark Amery Aug 15 '16 at 11:26
Nice solution! I's worth pointing out thought that you shouldn't use `shell=True` lightly as also mentioned in the documentation at https://docs.python.org/2/library/subprocess.html#frequently-used-arguments and maybe it also has a negative effect on RAM usage (two forks instead of one perhaps?). Also you might want to catch `OSError`s. – phk Oct 01 '16 at 19:28
Do you have any proof that there is significant overhead on this extra copy ? AFAIK the Copy-on-write should avoid this overhead. – Mohit May 12 '17 at 17:22
Well, if I do it the regular way, os.fork() fails with an out of memory error (even when executing tiny executables like echo), if I do it the way I did it now, it has no problems at all, even when executing larger executables. – Dakkaron May 12 '17 at 17:43
Great solution! I ran into an issue when "command" raised an exception, causing this cryptic error message: "TypeError: ('__init__() takes at least 3 arguments (1 given)', , ())". This was due to Python being unable to correctly pickle the exception object in the "result = e" line, causing "outQueue.get()" to throw the above error. The solution was to use the workaround in https://bugs.python.org/issue9400#msg274638, since the bug fix was not backported to Python 2.7 – GSTurtle Sep 14 '18 at 00:17

subprocess.check_output without high memory usage

1 Answers1