Bash: how to print and run a cmd array which has the pipe operator, |, in it

Question

This is a follow-up to my question here: How to write bash function to print and run command when the command has arguments with spaces or things to be expanded

Suppose I have this function to print and run a command stored in an array:

# Print and run the cmd stored in the passed-in array
print_and_run() {
    echo "Running cmd:  $*"
    # run the command by calling all elements of the command array at once
    "$@"
}

This works fine:

cmd_array=(ls -a /)
print_and_run "${cmd_array[@]}"

But this does NOT work:

cmd_array=(ls -a / | grep "home")
print_and_run "${cmd_array[@]}"

Error: syntax error near unexpected token `|':

eRCaGuy_hello_world/bash$ ./print_and_run.sh 
./print_and_run.sh: line 55: syntax error near unexpected token `|'
./print_and_run.sh: line 55: `cmd_array=(ls -a / | grep "home")'

How can I get this concept to work with the pipe operator (|) in the command?

Answer 1

If you want to treat an array element containing only | as an instruction to generate a pipeline, you can do that. I don't recommend it -- it means you have security risk if you don't verify that variables into your string can't consist only of a single pipe character -- but it's possible.

Below, we create a random single-use "$pipe" sigil to make that attack harder. If you're unwilling to do that, change [[ $arg = "$pipe" ]] to [[ $arg = "|" ]].

# generate something random to make an attacker's job harder
pipe=$(uuidgen)

# use that randomly-generated sigil in place of | in our array
cmd_array=(
  ls -a /
  "$pipe" grep "home"
)

exec_array_pipe() {
  local arg cmd_q
  local -a cmd=( )
  while (( $# )); do
    arg=$1; shift
    if [[ $arg = "$pipe" ]]; then
      # log an eval-safe copy of what we're about to run
      printf -v cmd_q '%q ' "${cmd[@]}"
      echo "Starting pipeline component: $cmd_q" >&2
      # Recurse into a new copy of ourselves as a child process
      "${cmd[@]}" | exec_array_pipe "$@"
      return
    fi
    cmd+=( "$arg" )
  done
  printf -v cmd_q '%q ' "${cmd[@]}"
  echo "Starting pipeline component: $cmd_q" >&2
  "${cmd[@]}"
}

exec_array_pipe "${cmd_array[@]}"

See this running in an online sandbox at https://ideone.com/IWOTfO

Answer 2

Do this instead. It works.

print_and_run() {
    echo "Running cmd: $1"
    eval "$1"
}

Example usage:

cmd='ls -a / | grep -C 9999 --color=always "home"'
print_and_run "$cmd"

Output:

Running cmd: ls -a / | grep -C 9999 --color=always "home"
(rest of output here, with the word "home" highlighted in red)

Answer 3

The general direction is that you don't. You do not store the whole command line to be printed later, and this is not the direction you should take.

The "bad" solution is to use eval.

The "good" solution is to store the literal '|' character inside the array (or some better representation of it) and parse the array, extract the pipe parts and execute them. This is presented by Charles in the other amazing answer. It is just rewriting the parser that already exists in the shell. It requires significant work, and expanding it will require significant work.

The end result is, is that you are reimplementing parts of shell inside shell. Basically writing a shell interpreter in shell. At this point, you can just consider taking Bash sources and implementing a new shopt -o print_the_command_before_executing option in the sources, which might just be simpler.

However, I believe the end goal is to give users a way to see what is being executed. I would propose to approach it like .gitlab-ci.yml does with script: statements. If you want to invent your own language with "debug" support, do just that instead of half-measures. Consider the following YAML file:

- ls -a / | grep "home"
- echo other commands
- for i in "stuff"; do
      echo "$i";
  done
- |
  for i in "stuff"; do
      echo "$i"
  done

Then the following "runner":

import yaml
import shlex
import os
import sys

script = []
input = yaml.safe_load(open(sys.argv[1], "r"))
for line in input:
    script += [
        "echo + " + shlex.quote(line).replace("\n", "<newline>"),  # some unicode like ␤ would look nice
        line,
    ]
os.execvp("bash", ["bash", "-c", "\n".join(script)])

Executing the runner results in:

+ ls -a / | grep "home"
home
+ echo other commands
other commands
+ for i in "stuff"; do echo "$i"; done
stuff
+ for i in "stuff"; do<newline>    echo "$i"<newline>done<newline>
stuff

This offers greater flexibility and is rather simple, supports any shell construct with ease. You can try gitlab-ci/cd on their repository and read the docs.

The YAML format is only an example of the input format. Using special comments like # --- cut --- between parts and extracting each part with the parser will allow running shellcheck over the script. Instead of generating a script with echo statements, you could run Bash interactively, print the part to be executed and then "feed" the part to be executed to interactive Bash. This will alow to preserve $?.

Either way - with a "good" solution, you end up with a custom parser.

Answer 4

Instead of passing an array, you can pass the whole function and use the output of declare -f with some custom parsing:

print_and_run() {
    echo "+ $(
        declare -f "$1" |
        # Remove `f() {` and `}`. Remove indentation.
        sed '1d;2d;$d;s/^ *//' |
        # Replace newlines with <newline>.
        sed -z 's/\n*$//;s/\n/<newline>/'
    )"
    "$@"
}

cmd() { ls -a / | grep "home"; }
print_and_run cmd

Results in:

+ ls --color -F -a / | grep "home"
home/

It will allow for supporting any shell construct and still allow you to check it with shellcheck and doesn't require that much work.