Python PIPE to popen stdin

Question

I am attempting something very similar to real time subprocess.Popen via stdout and PIPE

I, however, want to send input to the running process as well.

If I start a process in a separate thread using

process = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

I can send input using the terminal.

How would I send input from another source such as separate function not in the thread?

I cannot use Popen.communicate as the running process will never finish, as I am attempting real time interaction with the program.

Thanks in advance.

Here is my complete code, I am looking to have input sent to the subprocoess process when the send button is clicked.

from Tkinter import *`
from ttk import *`
import subprocess
from threading import Thread

class Example(Frame):

    def __init__(self, parent):
       Frame.__init__(self, parent)   

        self.parent = parent
        self.initUI()


    def initUI(self):    

        self.parent.title("Test Client")
        self.style = Style()
        self.style.theme_use("default")
        self.pack(fill=BOTH, expand=1)

        #Label, doesnt change
        lbl = Label(self, text="Client:")
        lbl.grid(row=0, column=1, sticky=W )

        #when output from client is shown
        global display
        display = Text(self,width=50,height=20)
        display.grid(row=1, column=1, sticky=E+W+N+S)

        #where user input is taken
        global prompt
        prompt = Entry(self,width=50)
        prompt.grid(row=3, column=1, sticky=E+W+N+S)

        #Button that will send input to client
        send = Button(self,text="Send",command=self.send)
        send.grid(row=3, column=2, sticky=N)
        get = Button(self,text="Get",command=self.get)
        get.grid(row=2, column=2, sticky=S)

    def get(self):
        print foo

    def send(self):
        sent = prompt.get()


def MyThread():
     global sent
     sent = 2
     cmd = ['nc', '-l', '-p', '50000']

     process = subprocess.Popen(cmd,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    while True:
        out = process.stdout.read(1)
        if out == '' and process.poll() != None:
            break
        if out != '':
            display.insert(INSERT, out)
            sys.stdout.write(out)
            sys.stdout.flush()

def main():
    root = Tk()
    root.geometry("500x410+300+300")
    app = Example(root)

    thread = Thread(target = MyThread, args=())
    thread.start()

    root.mainloop()

if __name__ == '__main__':
    main()  

Answer 1

First, you obviously need to add stdin=subprocess.PIPE to the Popen constructor, and then you can process.stdin.write just as you process.stdout.read.

But obviously, just as read can block if there's no data yet, write can block if the child isn't reading.

And even beyond the obvious, it's actually very hard to get the details right for using PIPEs in both directions with Popen to an interactive program without blocking anywhere. If you really want to do it, look at the source for communicate to see how it works. (There are known bugs before 3.2, so if you're on 2.x, you may have to do some backporting.) You will have to implement the code yourself, and if you want it to be cross-platform, you're going to have to do the whole mess that communicate does internally (spawning reader and writer threads for the pipes, etc.), and of course add another thread to not block the main thread on each attempt to communicate, and some kind of mechanism to message the main thread when the child is ready, and so on.

Alternatively, you can look at the various "async subprocess" projects on PyPI. The simplest one I know of today is async_subprocess, which basically just gives you a communicate that you can use without blocking.

Or, if you can use twisted (or possibly other event-based networking frameworks), there are wrappers around subprocess that plug into its event loop. (If you can wait for Python 3.4, or use the work-in-progress tulip on 3.3, someone's built something similar around tulip that may make it into 3.4.) And twisted even knows how to plug into Tkinter, so you don't have to manually handle two separate event loops and communicate between them.

If you only care about modern POSIX systems (not Windows), you can make it simpler by just putting the pipes in non-blocking mode and writing your code as if you were dealing with sockets.

But the easiest solution is probably to use something like pexpect instead of trying to script it manually. (As J.F. Sebastian points out, pexpect is Unix-only, but you can use a wrapper around pexpect for Unix and winpexpect for Windows.)

Answer 2

The select module in the standard library is made for these kind of situation:

process = subprocess.Popen(cmd,stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

while True:
   reads,writes,excs = select.select([process.stdout, process.stderr], [process.stdin], [], 1)
   for r in reads:
       out = r.read(1)
       display.insert(INSERT, out)
       sys.stdout.write(out)
       sys.stdout.flush()
   for w in writes:
       w.write('a')

You can pass a list of file objects or file descriptors to select() which will return those files that are have data ready for read/write or until an optional timeout.

The select module works on both Windows and Unix-like systems (Linux, Macs, etc).

Answer 3

A simple portable solution in your case with minimal code changes might be to create a writer thread that get items from a queue and writes them to the process' stdin and then put values into the queue whenever the button is pressed:

from subprocess import Popen, PIPE, STDOUT
from Queue import Queue

class Example(Frame):
    def __init__(self, parent, queue):
       # ...
       self.queue = queue
    # ...
    def send(self): # should be call on the button press
        self.queue.put(prompt.get())

def writer(input_queue, output): 
    for item in iter(input_queue.get, None): # get items until None found
        output.write(item)
    output.close()

def MyThread(queue):
    # ...
    #NOTE: you must set `stdin=PIPE` if you wan't to write to process.stdin
    process = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
    Thread(target=writer, args=[queue, process.stdin]).start()
    # ...

def main():
    # ...
    queue = Queue()
    app = Example(root, queue)
    Thread(target=MyThread, args=[queue]).start()
    # ...
    root.mainloop()
    queue.put(None) # no more input for the subprocess