Executing multiple commands using Popen.stdin

Question

I'd like to execute multiple commands in a standalone application launched from a python script, using pipes. The only way I could reliably pass the commands to the stdin of the program was using Popen.communicate but it closes the program after the command gets executed. If I use Popen.stdin.write than the command executes only 1 time out of 5 or so, it does not work reliable. What am I doing wrong?

To elaborate a bit :

I have an application that listens to stdin for commands and executes them line by line. I'd like to be able to run the application and pass various commands to it, based on the users interaction with a GUI. This is a simple test example:

import os, string
from subprocess import Popen, PIPE

command = "anApplication" 
process = Popen(command, shell=False, stderr=None, stdin=PIPE)

process.stdin.write("doSomething1\n")
process.stdin.flush()
process.stdin.write("doSomething2\n")
process.stdin.flush()

I'd expect to see the result of both commands but I don't get any response. (If I execute one of the Popen.write lines multiple times it occasionally works.)

And if I execute:

process.communicate("doSomething1")

it works perfectly but the application terminates.

Answer 1

If I understand your problem correctly, you want to interact (i.e. send commands and read the responses) with a console application.

If so, you may want to check an Expect-like library, like pexpect for Python: http://pexpect.sourceforge.net

It will make your life easier, because it will take care of synchronization, the problem that ddaa also describes. See also: http://www.noah.org/wiki/Pexpect#Q:_Why_not_just_use_a_pipe_.28popen.28.29.29.3F

Answer 2

Your code in the question should work as is. If it doesn't then either your actual code is different (e.g., you might use stdout=PIPE that may change the child buffering behavior) or it might indicate a bug in the child application itself such as the read-ahead bug in Python 2 i.e., your input is sent correctly by the parent process but it is stuck in the child's internal input buffer.

The following works on my Ubuntu machine:

#!/usr/bin/env python
import time
from subprocess import Popen, PIPE

LINE_BUFFERED = 1

#NOTE: the first argument is a list
p = Popen(['cat'], bufsize=LINE_BUFFERED, stdin=PIPE,
          universal_newlines=True)
with p.stdin:
    for cmd in ["doSomething1\n", "doSomethingElse\n"]:
        time.sleep(1) # a delay to see that the commands appear one by one
        p.stdin.write(cmd)
        p.stdin.flush() # use explicit flush() to workaround
                        #   buffering bugs on some Python versions
rc = p.wait()

Answer 3

The real issue here is whether the application is buffering its output, and if it is whether there's anything you can do to stop it. Presumably when the user generates a command and clicks a button on your GUI you want to see the output from that command before you require the user to enter the next.

Unfortunately there's nothing you can do on the client side of subprocess.Popen to ensure that when you have passed the application a command the application is making sure that all output is flushed to the final destination. You can call flush() all you like, but if it doesn't do the same, and you can't make it, then you are doomed to looking for workarounds.

Answer 4

It sounds like your application is treating input from a pipe in a strange way. This means it won't get all of the commands you send until you close the pipe.

So the approach I would suggest is just to do this:

process.stdin.write("command1\n")
process.stdin.write("command2\n")
process.stdin.write("command3\n")
process.stdin.close()

It doesn't sound like your Python program is reading output from the application, so it shouldn't matter if you send the commands all at once like that.