How to avoid zombie process with envoy.connect?

Question

I just discovered the 'envoy' module, a wrapper for python subprocess made by the creator of requests.

I have a problem with the 'connect' function: everytime I use it, it results in a zombie process and I can't get the status_code or the result.

c=envoy.connect("ls -al")
c.status_code == None
 True

If I do a 'ps -ef|grep thepid', I get a 'defunct' pid.

I can kill the zombie by doing a os.wait() or a c._process.wait(), but I can't get the result (stdout) of my command...

Any idea?

After a deeper look at my problem, I think it's platform/pythonVersion related: I'm using python 2.6 on Solaris10 (x86) and event the testcases given with envoy doesn't work. c = envoy.connect("sleep 1") time.sleep(2) self.assertEqual(c.status_code, 0) ==> Keeps having a zombie proc and a 'None' status_code. Unfortunately, I have no way to upgrade python, since it depends on an other team. I'm doing some tests on a linux machine... — SuperPython, Dec 05 '13 at 08:54

LiMar · Answer 1 · 2013-12-02T15:10:30.407

1

In your case you should use the run() method

As envoy documentation recommends:

r = envoy.run(cmd)
print r.status_code, r.std_out

But in case you want the command run asynchronously you can use connect() followed by block()

Once block() is called, return code becomes available. But block() would block your program so the logic should be.

c1=envoy.connect(cmd1)
c2=envoy.connect(cmd2)
c3=envoy.connect(cmd3)
... more staff ...
c1.block()
print c1.status_code

edited Dec 02 '13 at 15:10

answered Dec 02 '13 at 14:41

LiMar

2,822
3
22
28

run() works fine, but I wanted to use connect() to have non-blocking processes. My script has to launch tenth of commands. The 'ls -al' was just here for example purpose. The actual command take almost one minute to send a result... – SuperPython Dec 02 '13 at 14:49
Yes, I understand. Fixed the answer, look if it helps. – LiMar Dec 02 '13 at 15:11
1

+1 for `.block()`. `del c1` should also lead to reaping the zombie process. – jfs Dec 04 '13 at 00:13

score 1 · Answer 2 · answered Dec 04 '13 at 00:12

but I can't get the result (stdout) of my command...

ConnectedCommand (the type returned by envoy.connect()) doesn't seem to be ready. In particular, the command can deadlock if it receives/generates enough (depending on platform) input/output.

In addition to calling c.block() that also works for alive processes; you could remove all references to the command and use del c to reap the zombie. If subprocesses are not dead; they won't be reaped until a cleanup method is run on start of the next subprocess (it depends on the implementation).

If envoy.run() capabilities are not enough for your task; you could use subprocess module directly. For example, to pass input to several processes and collect corresponding results; you could use ThreadPool and .communicate() method:

#!/usr/bin/env python
from multiprocessing.pool import ThreadPool
from subprocess import Popen, PIPE

def process(child_input):
    child, input = child_input # unpack arguments
    return child.communicate(input)[0], child.returncode # get the result


# define input to be pass to subprocesses
params = b"a b c".split()

# start subprocesses (the command is just an example, use yours instead)
children = [Popen("( echo -n {0}; sleep {0}; cat )".format(len(params) - i),
                  shell=True, stdin=PIPE, stdout=PIPE)
            for i in range(len(params))]

# use threads to wait for results in parallel
pool = ThreadPool(len(params))
for output, returncode in pool.imap_unordered(process, zip(children, params)):
    if returncode == 0:
       print("Got %r" % output)

The less child sleeps the sooner its result is available to the parent regardless of the original order.

No zombies and It won't deadlock if input/output exceed pipe buffers.

How to avoid zombie process with envoy.connect?

2 Answers2