How to print stdout before writing stdin using subprocess module in Python

Question

I am writing a script in which in the external system command may sometimes require user input. I am not able to handle that properly. I have tried using os.popen4 and subprocess module but could not achieve the desired behavior.

Below mentioned example would show this problem using "cp" command. ("cp" command is used to show this problem, i am calling some different exe which may similarly prompt for user response in some scenarios). In this example there are two files present on disk and when user tries to copy file1 to file2, an conformer message comes up.

proc = subprocess.Popen("cp -i a.txt b.txt", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,)
stdout_val, stderr_val = proc.communicate()
print stdout_val

b.txt?

proc.communicate("y")

Now in this example if i read only stdout/stderr and prints it, later on if i try to write "y" or "n" based on user's input, i got an error that channel is closed.

Can some one please help me on achieving this behavior in python such that i can print stdout first, then should take user input and write stdin later on.

I found another solution (Threading) from Non-blocking read on a subprocess.PIPE in python , not sure whether it would help. But it appears it is printing question from cp command, i have modified code but not sure on how to write in threading code.

import sys
from subprocess import PIPE, Popen
from threading import Thread
try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

 p = Popen(['cp', '-i', 'a.txt', 'b.txt'],stdin=PIPE, stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.start()

try:
    line = q.get_nowait()
except Empty:
    print('no output yet')
else:
   pass

Do you want to allow the user to forward a response to the pipe or just have it say "y" no matter what? — jdi, Jul 23 '12 at 05:25
I want to interpret the user response myself and then code itself should communicate with the pipe. So user may type "y" or "n" or anything. — sarbjit, Jul 23 '12 at 06:50
the threading code won't help if the issue is ["block-buffering"](http://www.noah.org/wiki/Pexpect#Q:_Why_not_just_use_a_pipe_.28popen.28.29.29.3F) on the child's side i.e., your main thread won't block but you won't see any output in the queue. Try [`pexpect`, `winpexpect`, `wexpect`](http://stackoverflow.com/a/1042975/4279) as @ecatmur suggested. What is your child exe? Can you change its buffering settings? — jfs, Jul 24 '12 at 14:59
Child exe is IBM cleartool, i don't think i can change its buffering. I'll try winpexpect etc. — sarbjit, Jul 25 '12 at 03:30

ecatmur · Answer 1 · 2012-07-23T13:30:10.767

1

Popen.communicate will run the subprocess to completion, so you can't call it more than once. You could use the stdin and stdout attributes directly, although that's risky as you could deadlock if the process uses block buffering or the buffers fill up:

stdout_val = proc.stdout.readline()
print stdout_val
proc.stdin.write('y\n')

As there is a risk of deadlock and because this may not work if the process uses block buffering, you would do well to consider using the pexpect package instead.

edited Jul 23 '12 at 13:30

answered Jul 23 '12 at 13:15

ecatmur

152,476
27
293
366

Well, i tried this code but it just got hang while executing proc.stdout.readline() – sarbjit Jul 23 '12 at 13:23
@sarbjit yes, it's probably using block buffering. Try using pexpect. – ecatmur Jul 23 '12 at 13:29
pexpect doesn't works on windows. I want my code to be portable on windows and linux. – sarbjit Jul 24 '12 at 04:51
`wexpect` is a windows port of pexpect: http://stackoverflow.com/questions/1042778/can-i-use-expect-on-windows-without-installing-cygwin – ecatmur Jul 24 '12 at 09:19

score 0 · Answer 2 · answered Jul 23 '12 at 05:32

I don't have a technical answer to this question. More of just a solution. It has something to do with the way the process waits for the input, and once you communicate with the process, a None input is enough to close the process.

For your cp example, what you can do is check the return code immediately with proc.poll(). If the return value is None, you might assume it is trying to wait for input and can ask your user a question. You can then pass the response to the process via proc.communicate(response). It will then pass the value and proceed with the process.

Maybe someone else can chime in with a more technical reason why an initial communicate with a None value closes the process.

How to print stdout before writing stdin using subprocess module in Python

2 Answers2