Interactively run multiple commands from input() with subprocess

Question

I am looking for a way to create some sort of terminal "clone" in Python 3.6 using subprocess. The clone should behave just like the real terminal. The goal is to have a python script simulating a shell which behaves as much as the normal shell as possible. Including commands like cd or variable declarations.

My target system is Linux with gnome shell, but my problems are probably cross OS relate. At first I din't think that was too hard as you can easily run terminal commands using subprocess but I encountered some problems.

What I don't want to do:

#!/usr/bin/env python
import subprocess

while True:
    command = input(" >>> ").rstrip('\n')
    if command == 'quit':
        break
    subprocess.run(command, shell=True)

There would be a very easy way to run commands one after each other. The problem with that is, that this will start a new process for every command. So if I do the following commands it doesn't work as I want to:

 >>> ls
stackoverflow_subprocess.py
 >>> cd ..
 >>> ls
stackoverflow_subprocess.py

Because we start a new process every time, commands like cd do not have any effect. That's why I want to run all commands in the same process.

First Attempt:

#!/usr/bin/env python
from subprocess import PIPE, Popen

pipe = Popen("/bin/bash", stdin=PIPE, stdout=PIPE, stderr=PIPE)

quit_command = "quit"
while True:
    command = input(" >>> ")
    if command == quit_command:
        break
    command = str.encode(command + "\n")
    out, err = pipe.communicate(command)
    print(out,err)

This was my first attempt at solving my problem. This is what i got:

 >>> echo hi
b'hi\n' b''
 >>> echo hello
Traceback (most recent call last):
  File "/home/user/Python/Stackoverflow/subprocess.py", line 11, in <module>
    out, err = pipe.communicate(command)
  File "/usr/lib/python3.6/subprocess.py", line 818, in communicate
    raise ValueError("Cannot send input after starting communication")
ValueError: Cannot send input after starting communication

Process finished with exit code 1

So I can't just write multiple commands like this.

Second Attempt:

#!/usr/bin/env python
from subprocess import PIPE, Popen

fw = open("tmpout", "wb")
fr = open("tmpout", "r")

pipe = Popen("/bin/bash", stdin=PIPE, stdout=fw, stderr=fw, bufsize=1)

quit_command = "quit"
while True:
    command = input(" >>> ")
    if command == quit_command:
        break
    command = str.encode(command + "\n")
    pipe.stdin.write(command)
    out = fr.read()
    print(out)

This attempt was based on another stackoverflow question which was similar to mine: Interactive input/output using python

 >>> echo hi

 >>> echo hello 

 >>> quit

Process finished with exit code 0

However, this did not work as well. out was just an empty string. When I looked into it i realized, that the content of tmpout does not get written to the file until the program finished. Even if you close and reopen fw between each iteration it still just writes to tmpout after the program finishes.

Contents of tmpout after program finished:

hi
hello

Third Attempt:

#!/usr/bin/env python
from subprocess import PIPE, Popen

import os
import fcntl
from subprocess import Popen, PIPE


def setNonBlocking(fd):
    """
    Set the file description of the given file descriptor to non-blocking.
    """
    flags = fcntl.fcntl(fd, fcntl.F_GETFL)
    flags = flags | os.O_NONBLOCK
    fcntl.fcntl(fd, fcntl.F_SETFL, flags)


p = Popen("/bin/bash", stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=1)
setNonBlocking(p.stdout)
setNonBlocking(p.stderr)

quit_command = "quit"
while True:
    command = input(" >>> ")
    if command == quit_command:
        break
    command = str.encode(command + "\n")

    p.stdin.write(command)
    while True:
        try:
            out = p.stdout.read()
        except IOError:
            continue
        else:
            break
    out = p.stdout.read()
    print(out)

At last I tried the second solution from the Stackoverflow question mentioned above. This didn't work as well, as it just always returned None:

 >>> echo hi
None
 >>> echo hello
None
 >>> quit

Process finished with exit code 0

Question: Does anyone know a way to solve any of these issues? Is it possible to communicate more commands even after communication started? Or is it possible for the file to be written before the program ends? Or does anyone know how to get the actual output instead of just None for the last attempt?

Thank you in advance :)

Answer 1

Are you looking simply for this?

#!/usr/bin/env python

import subprocess

while True:
    command = input(" >>> ").rstrip('\n')
    if command == 'quit':
        break
    subprocess.run(command, shell=True)

As the following brief demo shows, it has some obvious flaws; but it certainly shows how to simply get out of the way by leaving the stdout and stderr of the subprocess alone.

 >>> echo hello
hello
 >>> foo=bar
 >>> echo "$foo"

 >>> # each command runs in a separate subshell
 >>> ^D
Traceback (most recent call last):
  File "/tmp/psh", line 6, in <module>
    command = input(" >>> ").rstrip('\n')
EOFError

Answer 2

Ok, so as it turns out, it is not that easy to run multiple command in the same process in order to keep shell builtins like cd or variable assignment. But They can be implemented yourself in python. So here is my version of an interactive Shell in Python using subprocess:

#!/usr/bin/env python

from subprocess import run
from os import path, curdir, chdir

home_dir = path.expanduser("~")


# Gets current directory and replaces your home directory with "~"
def current_dir():
    return path.abspath(curdir).replace(home_dir, "~")


# Escapes a string by replacing spaces " " with "\s" between quotation marks
def escape_space(string):
    out = ""
    quote = False
    for letter in string:
        quote = (quote != (letter == "\""))  # quote <- quote XOR letter is "
        if quote and letter == " ":
            letter = "\s"
        out += letter
    return out


# Dictionary that holds all variables
var_dict = {}


# Handles Variables
def handle_vars(command_args):
    for i in range(len(command_args)):
        arg = command_args[i]

        # Replace variables with their value
        if arg[0] == "$":
            if arg[1:] in var_dict:
                command_args[i] = var_dict[arg[1:]]
            else:
                command_args[i] = ""

        # Add new variable
        elif "=" in arg:
            arg_split = arg.split("=")
            var_dict[arg_split[0]] = arg_split[1]


quit_flag = False

if __name__ == "__main__":
    while True:
        display_dir = "\033[34m{}\033[39m$ ".format(current_dir())  # The current directory with color
        commands = input(display_dir).rstrip('\n').split(";")

        # Repeat for all commands (multiple commands are possible with ";")
        for cmd in commands:

            cmd = escape_space(cmd)
            command_args = cmd.split(" ")

            handle_vars(command_args)

            if command_args[0] == "quit":
                quit_flag = True
                break
            elif command_args[0] == "cd":
                chdir(command_args[1])  # change execution dir
            else:
                command = " ".join(command_args).replace("\s", " ")
                run(command, shell=True)

        if quit_flag:
            break

    print("Shell Terminated.")

This allows me to use cd, variables and also strings surrounded by double quotes ("). I hope this helps anyone who has a similar problem.