Python interactive shell not responding to input when run in subprocess

Question

I am making a terminal command line interface program as part of a bigger project. I want the user to be able to run arbitrary commands (like in cmd). The problem is that when I start a python process using subprocess, python doesn't write anything to stdout. I am not even sure if it reads what I wrote in stdin. This is my code:

from os import pipe, read, write
from subprocess import Popen
from time import sleep

# Create the stdin/stdout pipes
out_read_pipe_fd, out_write_pipe_fd = pipe()
in_read_pipe_fd, in_write_pipe_fd = pipe()

# Start the process
proc = Popen("python", stdin=in_read_pipe_fd, stdout=out_write_pipe_fd,
             close_fds=True, shell=True)

# Make sure the process started
sleep(2)

# Write stuff to stdin
write(in_write_pipe_fd, b"print(\"hello world\")\n")

# Read all of the data written to stdout 1 byte at a time
print("Reading:")
while True:
    print(repr(read(out_read_pipe_fd, 1)))

The code above works when I change "python" to "myexe.exe" where myexe.exe is my hello world program written in C++ compiled by MinGW. Why does this happen? This is the full code but the above example shows my problem. It also works correctly when I change "python" to "cmd".

PS: when I run python from the command prompt it gives me:

Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

That means that there should be stuff written to stdout.

Answer 1

The Problem

Note that you are connecting python to a non-tty standard input, and so it behaves differently from when you just run the command python in your terminal. It instead behaves as if you used the command cat script | python, which means it waits until stdin is closed, and then executes everything as a single script. This behavior is described in the docs:

The interpreter operates somewhat like the Unix shell: when called with standard input connected to a tty device, it reads and executes commands interactively; when called with a file name argument or with a file as standard input, it reads and executes a script from that file.

Try adding close(in_write_pipe_fd) before reading, and you'll see that it succeeds.

Solution 1: force python to run interactively

To solve your problem, we're gonna need python to ignore the fact it is not run interactively. When running python --help you might notice the flag -i:

-i     : inspect interactively after running script; forces a prompt even
         if stdin does not appear to be a terminal; also PYTHONINSPECT=x

Sounds good :) Just change your Popen call to:

Popen("python -i", stdin=in_read_pipe_fd, stdout=out_write_pipe_fd,
      close_fds=True, shell=True)

And stuff should start working as expected.

Solution 2: pretend to be a terminal

You might have heard of pty, a pseudo-terminal device. It is a feature in a few operating systems that allows you to connect a pipe to a tty driver instead of a terminal emulator, thus, in your case, allowing you write a terminal emulator yourself. You can use the python module pty to open one and connect it to the subprocess instead of a normal pipe. That will trick python to think it is connected to an actual tty device, and will also allow you to emulate Ctrl-C presses, arrow-up/arrow-down, and more.

But that comes with a price - some programs, when connected to a tty, will also alter their output accordingly. For example, in many linux distributions, the grep command colors the matched pattern in the output. If you don't make sure you can handle colors correctly in your program, or configure the tty to declare it doesn't support colors (and other tty features), you'll start getting garbage in some command's outputs.

Small note

I do feel like this might not be the best method to achieve your goal. If you describe it more in detail I might be able to help you think of an alternative :)

Answer 2

The python interpreter is more often used to run scripts from the command line than in interactive mode, therefore its interactive elements are not written to stdout else they would interfere with script output. Nobody wants to have to remove the introductory text from the script output.

To facilitate this, when interacting with the user, the interpreter uses the sys.displayhook method to deliberately send output to stdout otherwise nothing goes to stdout. The rest (e.g. the intro text, and >>> prompt) are written to stderr according to the docs:

• stdin is used for all interactive input (including calls to input());
• stdout is used for the output of print() and expression statements and for the prompts of input();
• The interpreter’s own prompts and its error messages go to stderr.