Continuous communication between parent and child subprocess in Python (Windows)?

Question

I have this script:

import subprocess

p = subprocess.Popen(["myProgram.exe"],
                     stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE)
while True:
    out, _ = p.communicate(input().encode())
    print(out.decode())

which works fine until the second input where I get:

ValueError: Cannot send input after starting communication

Is there a way to have multiple messages sent between the parent and child process in Windows ?

[EDIT]
I don't have access to the source code of myProgram.exe
It is an interactive command line application returning results from queries
Running >> myProgram.exe < in.txt > out.txt works fine with in.txt:

query1;
query2;
query3;

@Todd I tried it. But this only allows me to just send one message. — , Mar 14 '20 at 07:56
That's what the documentation of the function states. You can only send once. Maybe there's another way to interact using another method. — Todd, Mar 14 '20 at 08:03
Okay @entropyfeverone, I was able to mock a setup similar to your use case and get it to work. I updated the answer. Let me know how it goes. — Todd, Mar 14 '20 at 11:18

Todd · Accepted Answer · 2020-03-19T21:37:09.617

Interacting with another running process via stdin/stdout

To simulate the use case where a Python script starts a command line interactive process and sends/receives text over stdin/stdout, we have a primary script that starts another Python process running a simple interactive loop.

This can also be applied to cases where a Python script needs to start another process and just read its output as it comes in without any interactivity beyond that.

primary script

import subprocess
import threading
import queue
import time

if __name__ == '__main__':

    def enqueue_output(outp, q):
        for line in iter(outp.readline, ''):
            q.put(line)
        outp.close()

    q = queue.Queue()

    p = subprocess.Popen(["/usr/bin/python", "/test/interact.py"],
                         stdin    = subprocess.PIPE,
                         stdout   = subprocess.PIPE,
                         # stderr   = subprocess.STDOUT,
                         bufsize  = 1,
                         encoding ='utf-8')

    th = threading.Thread(target=enqueue_output, args=(p.stdout, q))
    th.daemon = True

    th.start()

    for i in range(4):

        print("dir()", file=p.stdin)

        print(f"Iteration ({i}) Parent received: {q.get()}", end='')
        # p.stdin.write("dir()\n")
        # while q.empty():
        #     time.sleep(0)
        # print(f"Parent: {q.get_nowait()}")

interact.py script

if __name__ == '__main__':

    for i in range(2):

        cmd = raw_input()

        print("Iteration (%i) cmd=%s" % (i, cmd))

        result = eval(cmd)

        print("Iteration (%i) result=%s" % (i, str(result)))

output

Iteration (0) Parent received: Iteration (0) cmd=dir()
Iteration (1) Parent received: Iteration (0) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i']
Iteration (2) Parent received: Iteration (1) cmd=dir()
Iteration (3) Parent received: Iteration (1) result=['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'cmd', 'i', 'result']

This Q&A was leveraged to simulate non-blocking reads from the target process: https://stackoverflow.com/a/4896288/7915759

This method provides a way to check for output without blocking in the main thread; q.empty() will tell you if there's no data. You can play around with blocking calls too using q.get() or with a timeout q.get(2) - the parameter is number of seconds. It can be a float value less than zero.

Text based interaction between processes can be done without the thread and queue, but this implementation gives more options on how to retrieve the data coming back.

The Popen() parameters, bufsize=1 and encoding='utf-8' make it possible to use <stdout>.readline() from the primary script and sets the encoding to an ascii compatible codec understood by both processes (1 is not the size of the buffer, it's a symbolic value indicating line buffering).

With this configuration, both processes can simply use print() to send text to each other. This configuration should be compatible for a lot of interactive text based command line tools.

No, it doesn't. I used ```input().encode()``` because I got error with just ```input()```, but again it does not print anything. — , Mar 14 '20 at 08:24
Is there an option from command line to do that ? I don't have access to the source code of ```myProgram.exe``` — , Mar 14 '20 at 08:27
No success :( I cannot understand why there are no examples like this. I mean, when I write in the command line ```>myProgram.exe < in.txt > out.txt``` it executes fine, reads commands line by line from ```in.txt``` and outputs to ```out.txt```. Yet, for some reason it is more difficult to have this behavior in a more streaming fashion using a python script. — , Mar 14 '20 at 08:38
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/209617/discussion-between-todd-and-entropyfeverone). — Todd, Mar 14 '20 at 08:40
I did, for some reason it only shows me once what I type. After that, I have to interrupt and then shows what I typed. — , Mar 14 '20 at 08:41
It's probably hanging on the readline() then.. so we may have to read it back into a buffer so it doesn't block. Click the chat link above if you want more help. — Todd, Mar 14 '20 at 08:43

Continuous communication between parent and child subprocess in Python (Windows)?

1 Answers1

Linked