0

Basically, I need to send bash commands that were generated in a script directly to the terminal from where it was run.

I tried using os.system and subprocess, however those methods don't work as I would expect for commands such as cd, or expanding ~ to a username automatically. I also don't want to resort to os.chdir, since the commands are generated, and I don't want to hardcode patterns if that makes sense.

In an ideal world, I could send any command directly to the terminal, like if Python was typing them into the terminal, rather than spawning a subprocess.

CorporalClegg
  • 11
  • 2
  • Here you will find an example of cd in python and why not to use https://stackoverflow.com/questions/21406887/subprocess-changing-directory – zodiac Mar 01 '23 at 20:03
  • What are you attempting to achieve with this? Can you give a short example? – Axe319 Mar 01 '23 at 20:06
  • 1
    If you want to support all possible shell commands, _you need a shell interpreter_ -- a long-lived one, not one that's started for each individual command and then exits when that command finishes (which is what you get with `os.system()` or most uses of `subprocess`). – Charles Duffy Mar 01 '23 at 20:12
  • @Axe319 I am writing a program that lets users control their terminal with natural language (using LLMs), sort of like a bash expert on demand. – CorporalClegg Mar 01 '23 at 20:14
  • 1
    LLMs aren't reliable enough to be used in that way. We've had questions, here, inside recent weeks, of the form "I just deleted my home directory by running code from ChatGPT; how can I fix it?". Unless you've got a sandbox that's preventing incorrect scripts from doing damage, running code from a LLM without an expert reviewing it ahead-of-time is an _extremely_ bad idea. – Charles Duffy Mar 01 '23 at 20:15
  • @CharlesDuffy Thanks for the insight. I do have access to sandbox VMs, I am writing my master's thesis on this subject, and commands are only run after approval. So I won't let it rm -rf * or fork bomb me :-) – CorporalClegg Mar 01 '23 at 20:19
  • 1
    Fair 'nuff. Mind, there are much more obscure bugs as well -- places where humans get shell behavior wrong, and LLMs trained on things humans write _also_ do. I've seen a LLM write code of the form `find /uploads -type f -name '*.png' -exec sh -c '...{}...'`, f/e, and that's an introduction for some joker to create a `/uploads/$(rm rf ~).png` file, causing the person trying to process their uploads folder to find their home directory deleted. – Charles Duffy Mar 01 '23 at 20:21
  • 1
    Anyhow -- if you want a long-lived interactive shell as a subprocess, _spawn one_. You can create a process with `subprocess.Popen(['bash', '-il'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)`, feed it commands on stdin, read the output from stdout and stderr. There are a bunch of messy complications in doing that, which is why you'll probably end up wanting to use a tool like `pexpect` to do the heavy lifting, but there's nothing stopping you. – Charles Duffy Mar 01 '23 at 20:27
  • 1
    You certainly can't run code in the parent-process shell that spawned you, but there's no reason to anyhow; a child-process interactive shell does just as well, and gives you more control (letting you, f/e, generate a unique UUID and configure it as the shell's prompt so you have an unambiguous marker as to when the shell is ready for new commands). – Charles Duffy Mar 01 '23 at 20:29
  • (I advise against talking about, or thinking about, terminals in this context unless you have a _very good reason_ to do so; real terminal emulation would make your life a lot harder, and for no good reason unless you want your LLM to drive curses applications like visual editors). – Charles Duffy Mar 01 '23 at 20:30
  • ...BTW, to get a rough idea of what I was talking about in referring to "messy complications", https://stackoverflow.com/questions/11457931/running-an-interactive-command-from-within-python might be a place to start. – Charles Duffy Mar 01 '23 at 20:35

2 Answers2

1

What you're asking for with cd is not possible. cd is a shell builtin meaning you have to launch a shell as a child process to run it, but children can't change their parent's working directory or anything else about the parent's environment. If you launch a shell with subprocess and run a cd command in it, the child changes its own working directory and then exits while the parent maintains its own working directory state the entire time.

Expanding ~ requires os.path.expanduser. Tilde expansion is not inherent to the OS or filesystem, it's something that shells typically implement (see Bash Tilde Expansion for example). Python has its own command for performing this expansion.

tjm3772
  • 2,346
  • 2
  • 10
  • Thank you for the elaborate response. Would there be a way to use a shell script, and somehow wrap my python code so that the shell script could handle running these commands it got from python? – CorporalClegg Mar 01 '23 at 20:12
  • 2
    @CorporalClegg, if you have a complete script (that both does all your setup like `cd` commands, and does all the operations that depend on that setup), `subprocess` with `shell=True` will handle that just fine (and we'd need a question showing a specific problem you're encountering if it doesn't work for you). It's the one-command-at-a-time situation that's a problem. – Charles Duffy Mar 01 '23 at 20:13
  • `os.path.expanduser` is about *Python* doing tilde expansion, not about the behavior of the shell in which `os.system()` or `subprocess.run(shell=True)` runs commands. – John Bollinger Mar 01 '23 at 20:23
1

In an ideal world, I could send any command directly to the terminal, like if Python was typing them into the terminal, rather than spawning a subprocess.

That makes sense only if by "terminal" you mean the shell from which your Python script was launched. But that shell will be waiting for Python to terminate before processing any more input, so the script cannot feed it commands to process. Or even if you started the script in the background, it still has no access to inject data that the parent shell would read as input.

But you can launch a new shell via subprocess.Popen(), send multiple commands to it over an extended period, and let it produce output to the standard output and standard error streams inherited from the parent shell. It would look something like this:

import subprocess

# Launch a subshell

shell = subprocess.Popen(['/bin/bash'], stdin=subprocess.PIPE)

# Send commands

shell.stdin.write(b'cd ~/Downloads\n')
shell.stdin.flush()

shell.stdin.write(b'ls\n')
shell.stdin.flush()

# Terminate

shell.stdin.write(b'exit\n')
shell.stdin.flush()
shell.wait(timeout=seconds_to_wait)

Note that like any subprocess, the shell's standard input is fed via a binary stream, not a text stream. Note also that you need to ensure that commands are terminated with a newline, and you will want to flush, as shown, after each one to ensure that the shell receives it right away.

There are other ways to terminate the subshell (Popen.terminate(), Popen.kill(), ...), and you may need to be flexible to avoid unwantedly terminating the shell while some process started by it is still running.

You will also want to exercise care if you launch interactive commands in the subshell, because like that shell itself, they will receive their input from the Python script, not directly from a user.

Additionally, do note that bash differentiates between interactive and non-interactive shells, and somewhat between login and non-login shells. This affects which startup files are read, if any, the default values for some shell options, and probably some other things that escape me at the moment. The above example uses a non-interactive,* non-login shell. That is purposeful, but it may be that you would be better served by an interactive shell, or even an interactive login shell. Those and other details can be controlled by passing additional command-line options to bash.

*... in Bash's rather specific sense of "non-interactive". That does not prevent it from receiving commands from its standard input.

John Bollinger
  • 160,171
  • 8
  • 81
  • 157
  • Thank you for the elaborate response, now I understand the problem with how I wanted it to work and I wish I'd paid a bit more attention in Operating Systems :D What would be your advice with respect to launching interactive commands? Do you have any resources I can look at? In my case the Python script is supposed to help the user with tasks they want to carry out on the shell, so having that interactive component would help. – CorporalClegg Mar 02 '23 at 14:29
  • @CorporalClegg, we generally consider requests for external resources to be off-topic here. I think that supporting interactive commands will require the script you are interposing between user and shell to be stateful, in the sense of discriminating between a command-helper mode and a plain pass-through mode. How it is triggered to switch between the two would be an interesting research problem. – John Bollinger Mar 02 '23 at 15:29