Python subprocess.Popen blocks with shell and pipe

Question

Running the following command: yes | head -1, from the shell, outputs y. Using python's subprocess module to call it with a shell hangs indefinitely:

import subprocess

arg = "yes | head -1"
process = subprocess.Popen(arg,
    shell=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

print "Command started: %d" % process.pid
r = process.communicate()
print "Command ended: %s %s" % r

Killing the process exogenously using kill does not help, nor does making the child process it's session leader using preexec_fn=os.setsid.

What could be causing this behavior, and is there anyway I can prevent it?

I'm running python 2.7.3 and my /bin/sh is GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin12)

I can't reproduce it on Python 2.7 on Ubuntu. It works ok +/- [the expected "write" error](http://stackoverflow.com/q/22077881/4279). — jfs, Apr 14 '14 at 18:23

score 1 · Accepted Answer · edited May 23 '17 at 12:27

So it turns out that this is caused due to a known issue with python not resetting signal handlers before exec in the subprocess module.

This is the same problem that causes 'yes' reporting error with subprocess communicate(), except with GNU yes, the EPIPE returned by a write causes the program to abort, whereas with BSD yes (which is used on OS X as far as I can tell), the return code of the write is not checked, so without a SIGPIPE, yes will not terminate.

It can be fixed by backporting the reset_signals code, as in the above linked question:

def restore_signals():
    signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ')
    for sig in signals:
        if hasattr(signal, sig):
            signal.signal(getattr(signal, sig), signal.SIG_DFL)

then setting preexec_fn=restore_signals in the Popen call.

Python subprocess.Popen blocks with shell and pipe

1 Answers1

Linked