How can I dynamically substitute array entries for an arbitrary command parameter?

Question

This question is inspired by bash nested variable in for loop.

If I have an array in bash, and I want to be able to run an arbitrary command for every element of that array, is there a way to do that via a generic function, as opposed to with a loop? That is:

dest_array=( host1:/foo host2:/bar host3:/baz )
copy ./file dest_array

and have each expansion called:

copy ./file host1:/foo
copy ./file host2:/bar
copy ./file host3:/baz

Even better, is there a way to do this for multiple arrays? For instance:

sources=( first second )
dests=( host1:/foo host2:/bar )
copy sources dests

invoking (in no particular order):

copy first host1:/foo
copy first host2:/bar
copy second host1:/foo
copy second host2:/bar

Charles Duffy · Accepted Answer · 2017-05-12T00:39:13.917

2

Consider the following function, written for bash 4.3 or later:

run_for_each() {
  local -n _items=$1; shift
  local sigil=$1; shift
  local -a args=( "$@" )
  local -a call
  local retval=0
  for item in "${_items[@]}"; do
    call=( "${args[@]//$sigil/$item}" )
    "${call[@]}" || (( retval |= $? ))
  done
  return "$retval"
}

As an example of usage:

sources=( first second )
dests=( host1:/foo host2:/bar )

run_for_each sources SOURCE \
  run_for_each dests DEST \
    rsync -Pv SOURCE DEST

If you wanted to make it concurrent, that might look like:

run_for_each_concurrent() {
  local -n _items=$1; shift
  local sigil=$1; shift
  local -a args=( "$@" )
  local -a pids=( )
  local -a call
  local retval=0
  for item in "${_items[@]}"; do
    call=( "${args[@]//$sigil/$item}" )
    "${call[@]}" & pids+=( "$!" )
  done
  for pid in "${pids[@]}"; do
    wait "$pid" || (( retval |= $? ))
  done
  return "$retval"
}

...which will run one process per array entry, all at the same time; wait for them all to exit; and return the ORed-together exit status of all those subprocesses.

Portability Modifications (Adapting for Bash 3.2 Compatibility)

By the way -- if you don't have bash 4.3, the above can be made to work with older releases by replacing this line:

local -n _items=$1; shift

with the following instead:

printf -v cmd 'local -a _items=( "${%q[@]}" )' "$1" && eval "$cmd"; shift

edited May 12 '17 at 00:39

answered May 11 '17 at 22:02

Charles Duffy

280,126
43
390
441

Would plain `sources` work without doing `"${sources[*]}"`? – l'L'l May 11 '17 at 22:29
@l'L'l, you're accusing me of `${sources[*]}`?! There's nothing of the sort in the above -- take a closer read. `local -n` makes `_items` a nameref, so it's referring to the variable named by `$1`. – Charles Duffy May 11 '17 at 22:30
Gotcha. Shellcheck is missing the indirect access, in this case. (I used "accusing" because `"${array[*]}"` makes individual elements indistinguishable in the general case, so it's a practice I wouldn't use if dealing with unknown data). – Charles Duffy May 11 '17 at 22:31
Yeah I figured; and I didn't look at the rest closely enough to know how you were referencing those :) – l'L'l May 11 '17 at 22:32
One more question, what does `local -n` do? specifically the `-n`, as it returned `local: -n: invalid option` on GNU bash, version 3.2.57. – l'L'l May 11 '17 at 22:51
@l'L'l, defines a nameref; requires 4.3 or newer (as specified in the first paragraph). – Charles Duffy May 12 '17 at 00:32
1

@l'L'l, ...I've added an addendum describing a modification to support bash 3.2. Note that copying-and-pasting it into an interactive shell might not quite work right unless history expansion is turned off, as with `set +H`. – Charles Duffy May 12 '17 at 00:42
Not that I have a better way, but the answer goes against the brief which asks for a solution without loops. In the end have you really gained anything outside of using referenced items? – grail May 12 '17 at 05:10
@grail, the loops are encapsulated -- from the caller's perspective, it's just a function call, which is the point. (If you check the other question this was derived from, the goal was function encapsulation -- the inner function using a loop was explicitly OK). – Charles Duffy May 12 '17 at 15:20

score 1 · Answer 2 · answered Dec 18 '17 at 17:47

What do you think about this solution:

run_for() {
   local -n _sources=$1; shift
   local src_label=$1; shift
   local -n _dests=$1; shift
   local dst_label=$1; shift

   local -a cmd=( "$@" ) execute
   local retval=0

   for src_item in "${_sources[@]}"; do
      for dst_item in "${_dests[@]}"; do
        execute=()
        for cmd_item in "${cmd[@]}"; do
           case $cmd_item in
              $src_label) execute+=("$src_item") ;;
              $dst_label) execute+=("$dst_item") ;;
                       *) execute+=("$cmd_item") ;;
           esac 
        done          
        "${execute[@]}" || (( retval |= $? ))
      done

   done

   return "$retval"
}

This solution also works with bash 4.3 or later and uses for loop 3 times (not really pretty). However, it has a more straightforward usage and handles the following case correctly:

sources=( first second DEST )
dests=( host1 host2 host3 )

run_for sources SOURCE dests DEST echo "<<" SOURCE - DEST ">>"

This solution cannot handle the following:

run_for sources SOURCE dests DEST echo aaSOURCE - DESTaa

Although, I would not really consider aaSOURCE to be a valid label. If it is, the following could, in my opinion, copy the behavior of your solution while still preserving the more straightforward usage. It will also have the same drawback when one of the sources is equal to $dst_label:

for src_item in "${_sources[@]}"; do

   tmp=("${cmd[@]//$src_label/$src_item}")    
   for dst_item in "${_dests[@]}"; do
      execute=("${tmp[@]//$dst_label/$dst_item}")
      "${execute[@]}" || (( retval |= $? ))
   done

done

Hmm. This is a little less generalized inasmuch as it assumes exactly two labels, as opposed to using a single label per invocation and letting invocations be nested as much as the user wants... but it's well-written, best-practices code in any event, and certainly adds something useful to the discussion. Thanks! :) — Charles Duffy, Dec 18 '17 at 17:56
@CharlesDuffy That is true, I really haven't considered multiple labels. Now I see the true potential of your solution! Anyway, very nice "question". — PesaThe, Dec 18 '17 at 18:03

How can I dynamically substitute array entries for an arbitrary command parameter?

2 Answers2

Portability Modifications (Adapting for Bash 3.2 Compatibility)

Linked