prevent unexpected stdin reads and lock in subprocess

Question

A simple case I'm trying to solve for all situations. I am running a subprocess for performing a certain task, and I don't expect it to ask for stdin, but in rare cases that I might not even expect, it might try to read. I would like to prevent it from hanging in that case.

here is a classic example:

import subprocess
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"])
p.wait()

This will hang forever. I have already tried adding

stdin=open(os.devnull)

and such..

will post if I find a valuable solution. would be enough for me to receive an exception in the parent process - instead of hanging on communicate/wait endlessly.

update: it seems the problem might be even more complicated than I initially expected, the subprocess (in password and other cases) reads from other file descriptors - like the /dev/tty to interact with the shell. might not be as easy to solve as I thought..

score 5 · Accepted Answer · edited May 23 '17 at 12:31

If your child process may ask for a password then it may do it outside of standard input/output/error streams if a tty is available, see the first reason in Q: Why not just use a pipe (popen())?

As you've noticed, creating a new session prevents the subprocess from using the parent's tty e.g., if you have ask-password.py script:

#!/usr/bin/env python
"""Ask for password. It defaults to working with a terminal directly."""
from getpass import getpass

try:
    _ = getpass()
except EOFError:
    pass # ignore
else:
    assert 0

then to call it as a subprocess so that it would not hang awaiting for the password, you could use start_new_session=True parameter:

#!/usr/bin/env python3
import subprocess
import sys

subprocess.check_call([sys.executable, 'ask-password.py'],
                      stdin=subprocess.DEVNULL, start_new_session=True,
                      stderr=subprocess.DEVNULL)

stderr is redirected here too because getpass() uses it as a fallback, to print warnings and the prompt.

To emulate start_new_session=True on Unix on Python 2, you could use preexec_fn=os.setsid.

To emulate subprocess.DEVNULL on Python 2, you could use DEVNULL=open(os.devnull, 'r+b', 0) or pass stdin=PIPE and close it immediately using .communicate():

#!/usr/bin/env python2
import os
import sys
from subprocess import Popen, PIPE

Popen([sys.executable, 'ask-password.py'],
      stdin=PIPE, preexec_fn=os.setsid,
      stderr=PIPE).communicate() #NOTE: assume small output on stderr

Note: you don't need .communicate() unless you use subprocess.PIPE. check_call() is perfectly safe if you use an object with a real file descriptor (.fileno()) such as returned by open(os.devnull, ..). The redirection occurs before the child process is executed (after fork(), before exec()) -- there is no reason to use .communicate() instead of check_call() here.

Dani K · Answer 2 · 2015-10-24T08:20:19.950

3

Apparently the culprit is the direct usage of /dev/tty and such.

On linux at least, one solution is to add to the Popen call the following parameter:

preexec_fn=os.setsid

which causes a new session id to be set, and disallows reading from the tty directly. i will probably use the following code (stdin close is just in case):

import subprocess
import os
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"],
                     stdin=subprocess.PIPE, preexec_fn=os.setsid)
p.stdin.close() #just in case
p.wait()

last two lines can be replaced by one call:

p.communicate()

since communicate() closes stdin file after sending all the input supplied.

Simple and elegant it seems.

Alternatively:

import subprocess
import os
p = subprocess.Popen(["unzip", "-tqq", "encrypted.zip"],
                     stdin=open(os.devnull), preexec_fn=os.setsid)
p.communicate()

edited Oct 24 '15 at 08:20

answered Oct 22 '15 at 09:35

Dani K

280
1
9

unrelated: you could use `subprocess.DEVNULL` on Python 3. I like `os.setsid` solution more (if it works), but you could also provide a pseudo-tty using [`pexpect` or `pty.openpty()`](http://stackoverflow.com/a/12471855/4279) (your case is [the first reason described in `pexpect` docs](http://pexpect.readthedocs.org/en/latest/FAQ.html#whynotpipe)). – jfs Oct 22 '15 at 21:23
1

you can call `p.communicate()` instead of `p.stdin.close(); p.wait()` in the first example. There is no reason to call `p.communicate()` in the last example (just use `subprocess.check_call()` instead). – jfs Oct 22 '15 at 21:26
meta: you can [accept your own answer](http://stackoverflow.com/help/self-answer) – jfs Oct 22 '15 at 21:27
@J.F.Sebastian awesome, had to look at communicate source to see it closes stdin after writing input. wonderful, thanks! – Dani K Oct 24 '15 at 08:11
@J.F.Sebastian if i understand correctly, check_call is an option for when you don't want stdout/stderr from the process and just want notification if it failed. also a viable option - but with check_call i would have to use the setsid also I'm assuming. correct? – Dani K Oct 24 '15 at 08:19
you don't need to look at `.communicate()` source; its [behavior is documented](https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate): *"Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate."* -- it implies that `p.stdin` is closed otherwise you couldn't call a subprocess that may read an unlimited input. Your examples should be: `Popen([..], stdin=PIPE, preexec_fn=os.setsid).communicate()` and `call([..], stdin=DEVNULL, preexec_fn=os.setsid)` – jfs Oct 24 '15 at 09:49
Instead of using `preexec_fn`, you can set `start_new_session=True` (Python 3.2 and up). This triggers a `setsid()` call in the new child process without the need for a cryptic-looking `preexec_fn` argument. – Martijn Pieters Apr 04 '19 at 15:07
can't import setsid from os in python 3.7.9. any substitute? – Ratul Hasan Jan 13 '21 at 06:51

prevent unexpected stdin reads and lock in subprocess

2 Answers2

Linked