0

I am building a tool that in part acts as a preprocessor to yaml that should have shell command interpolation for getting outside information when parsing said yaml

The way in which I've built this is to find shell commands that are wrapped in a $eval(...) and use subprocess.check_output to get the return of such command.

I'm unable to figure out how to use $variables in check_outout, for example where export foo=bar, subprocess.check_output(["echo", "$foo"]) should return b'bar'. Instead, the literal b'$foo' is returned.

Is there a way to tell check_output to correctly parse these $variables?

gastrodon
  • 75
  • 3
  • 9
  • 1
    When you run `echo $foo` *in a shell*, that *shell* is what's responsible for replacing `$foo` with however many separate arguments it's supposed to be munged into by string-splitting and glob-expansion stops given the actual value of the relevant variable. Those steps are already completed before `echo` is ever started. – Charles Duffy Feb 04 '20 at 17:33
  • 1
    ...so, when you call `echo` *without* a shell, you can't expect `$foo` to be replaced with anything else; it remains `$foo`, because you didn't *run a shell*, and the shell is the only thing that performs that operation. (Also, you get `/bin/echo` instead of your shell's internal `echo` builtin implementation, so other behavior may subtly differ; see the APPLICATION USAGE and RATIONALE sections of [the POSIX `echo` specification](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html) describing just how loosely `echo`'s behavior is defined). – Charles Duffy Feb 04 '20 at 17:34
  • 1
    ...moral of this story: If you want a shell, you actually need to *run a shell*, by using `shell=True` and passing the entire string you want parsed by that shell as code as a single string. (If you pass a list with `shell=True`, the first element in the list is what it parses as code, and subsequent elements are arguments to that code -- which is to say, its `$0`, `$1`, etc). – Charles Duffy Feb 04 '20 at 17:39
  • @CharlesDuffy thanks, `shell = True` is the way that I've implemented this tool currently. I didn't know if there was a better way to do this. I was unaware that $var handling was the role of the shell, so that tells me that what I've already done is what I should stick with – gastrodon Feb 04 '20 at 17:42
  • 1
    Right. There are very good reasons not to run a shell with data you might not trust passed as code, but the very nature the thing you're trying to build is such that the tool is completely unusable if you *don't* trust the input to run arbitrary commands as whatever user the program runs as. – Charles Duffy Feb 04 '20 at 17:53
  • 2
    BTW, `shell=True` is not magic; it just replaces your arg with a single-item list encapsulating it should it be a string, then prepends the list `['sh', '-c']`. You could run `subprocess.check_output(["sh", "-c", "echo $foo"])` without `shell=True` as a substitute for `subprocess.check_output("echo $foo", shell=True)` and behavior would be 100% indistinguishable. – Charles Duffy Feb 04 '20 at 17:54

1 Answers1

0

If foo is a variable in your environment, you include that value directly in the list, not as a shell parameter to be expanded.

check_output(["echo", os.environ["foo"]])
chepner
  • 497,756
  • 71
  • 530
  • 681
  • 1
    Context, since this is not the OP's first question: They want the output of arbitrary shell commands. Why they're not willing to actually invoke a shell to get this is something of a mystery as yet. – Charles Duffy Feb 04 '20 at 17:37