Run continuous .exe file with Python

Question

I am trying to implement a python code where I need to run an external executable file. After some research I found subprocess module and tried it. However, when I run the code, the subprocess exits and returns a CompletedProcess object, which is not okay for me because the .exe I am working with is actually a continuous program where it takes input and generates output until it is closed. (My program will be sending its inputs and recieving the outputs.)

So this code fails.

import subprocess

proc = subprocess.run("some.exe", stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print(proc) # To see what it is.

It captures the programs first output in stdout but then quits, rather than expecting new input.

I am also having a problem with the terminology here, this executable I am using is capable of running commands while still being able to recieve new commands, is this called asynchronous or Non-blocking ?

To my question, what is the correct way of handling this ? There are a lot of questions in SO close to this but I could not be sure which were answering my problem or answering it in a modern way (There are very old questions).

It would appear subprocess module undergone some changes with Python 3.6.1 https://docs.python.org/3/library/subprocess.html

So it would be best if you could provide the solution in the newest way, or at least mention the newer way and how it operates.

I am working on Windows 10, Python 3.6.1

Edit1: I have come across subprocess.Popen, which may be what I need, but I am yet to discover how it is properly used.

Edit2: The module pexpect seems to be designed for these kind of problems, but I am not sure if using this going to make my time much easier. Any suggestions appreciated.

Answer 1

From the docs:

subprocess.run()

Run the command described by args. Wait for command to complete, then return a CompletedProcess instance.

Since your subprocess doesn't end after initially calling it, you cannot use run but need to use Popen instead.

You could use something like this:

import subprocess
p = subprocess.Popen(exe_path,
                     stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT)
while p.poll() is None:
    p.stdin.write(some_command)
    p.stdin.flush()
    out = p.stdout.readline()

But note that this can lead to a deadlock as described in the documentation:

Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

Unfortunately you cannot use communicate because this would require you to call the process with all input. I'd give it a try with the version above and see if the buffers really cause problems in your application.