Use subprocess.Popen non-interactively

Question

I'm trying to non-interactively execute a git pull command with subprocess.Popen. Specifically, if for some reason git asks for a password, the command should fail. But instead, it asks the password in the shell I run my python program in, and waits for my input, this not what I want.

I also need to capture the output of the command (this is why I use subprocess.Popen). I have noted that if I use stdout = DEVNULL then it behaves as I want, excepted I do need to pipe stdout to capture it.

Here is a simplified version of the code I use to execute the command:

from subprocess import Popen, PIPE, STDOUT, DEVNULL

process = Popen(['git', 'clone', 'https://some-url.git'], stdin = DEVNULL, stdout = PIPE)

for line in process.stdout:
    print(line.decode())

process.wait()

I've also tried with stdin = None. This parameter seems to have no effect.

I'm executing the python program in a bash console on a Debian 11 system (but I'd like a portable solution).

score 0 · Answer 1 · answered Aug 15 '22 at 11:38

0

Why are you redirecting errors to STDOUT if you don't want to show the output? For it to end up on the stdout filehandle of the process object, you want PIPE, but if you don't care what it is, just send it to DEVNULL.

Tangentially, you should avoid the bare Popen when what you want can be accomplished with subprocess.run or subprocess.check_output.

output = subprocess.check_output(
    ['git', 'clone', 'https://some-url.git'],
    # perhaps
    stdin=DEVNULL, stderr=DEVNULL)

shell=False is already the default, so you don't need to specify that.

check_output will raise an error if the git command fails; you might want to wrap that in try/except, or perhaps revert to subprocess.run which gives you more control over these details.

Tangentially, perhaps note that git may separately run ssh and generate error output from that e.g. when you connect to a host you have not connected to before. The sustainable solution to that is to specify options for how to handle the connection, perhaps in your .ssh/config.

answered Aug 15 '22 at 11:38

tripleee

175,061
34
275
318

For (a lot more) details, see also https://stackoverflow.com/questions/4256107/running-bash-commands-in-python/51950538#51950538 – tripleee Aug 15 '22 at 11:41
I want to capture stderr and stdout, and I want to do that in real time as lines are added to the output. This code is for a web server, if an admin is connected he will be able to see the output. But I don't want the process to hang waiting for an input, ever. In normal situations it won't ask for an input, but if someone misconfigures something it may, and this should be reported as an error instead of waiting for user input. – youen Aug 15 '22 at 12:27
The same basic guidance still applies; don't redirect stderr to the console if that's not what you want. The [tag:subprocess] tag has several links to questions about running subprocesses concurrently if you need more help with that part as well. – tripleee Aug 15 '22 at 12:28
I've updated the question. I've removed the stderr parameter, because it doesn't change the behavior. The problem is with `stdout = PIPE`. If I do that, the process waits for an input. If I don't, it doesn't wait and fails as expected, but then I can't capture the output, which I need to do. – youen Aug 15 '22 at 12:34
https://stackoverflow.com/a/44790480/874188 has a solution which captures both stdout and stderr but unfortunately it uses threads, which I suppose any sane solution should avoid. – tripleee Aug 15 '22 at 12:57
Thanks for the pointer, but this seems irrelevant to my problem. I think we can forget about stderr. My problem is that if I pipe stdout, then the subprocess tries to read stdin (and hangs forever as nothing will ever happen with stdin). – youen Aug 15 '22 at 15:34
Several of the other answers to the linked question purport to deal with stdout only. I would start with the one which tries to read a byte at a time, but I had to tend to other things before I could complete my experiments. I'll probably update this answer tomorrow. – tripleee Aug 15 '22 at 15:49

score 0 · Answer 2 · answered Aug 15 '22 at 18:56

I've found a way to deal with this issue, by using setsid to run the command in a non-interactive session. But this is a Linux thing, I'm still interested if there is a way to solve the issue with python in a portable way (I haven't made any test on Windows yet, maybe the issue doesn't exist there to begin with, but surely setsid won't work).

The code would look like this:

from subprocess import Popen, PIPE, STDOUT, DEVNULL

process = Popen(['setsid', 'git', 'clone', 'https://some-url.git'], stdout = PIPE)

for line in process.stdout:
    print(line.decode())

process.wait()

Use subprocess.Popen non-interactively

2 Answers2