bash: How do I ensure termination of process substitution used with exec?

Question

If I run

$#/bin/bash
for i in `seq 5`; do
    exec 3> >(sed -e "s/^/$i: /"; echo "$i-")
    echo foo >&3
    echo bar >&3
    exec 3>&-
done

then the result is not synchronous; it could be something like:

1: foo
1: bar
2: foo
2: bar
1-
3: foo
3: bar
2-
3-
4: foo
5: foo
4: bar
5: bar
4-
5-

How do I ensure that the process substitution >(...) is completed before proceeding to the next iteration?

Inserting sleep 0.1 after exec 3>&- helped, but it's inelegant, inefficient, and not guaranteed to always work.

EDIT: The example may look silly, but it was for illustration only. What I'm doing is reading a stream of input in a loop, feeding each line to a process which occasionally changes during the loop. Easier explained in code:

# again, simplified for illustration
while IFS= read line; do
    case $line in
    @*)
        exec 3>&-
        filename=${line:1}
        echo "starting $filename"
        exec 3> >(sort >"$filename"; echo "finished $filename")
        ;;
    *)
        echo "$line" >&3
        ;;
    esac
done
exec 3>&-

Answer 1

The following works in bash 4, using coprocesses:

#!/bin/bash
fd_re='^[0-9]+$'
cleanup_and_wait() {
    if [[ ${COPROC[1]} =~ $fd_re ]] ; then
        eval "exec ${COPROC[1]}<&-"
        echo "waiting for $filename to finish" >&2
        wait $COPROC_PID
    fi
}

while IFS= read -r line; do
    case $line in
    @*)
        cleanup_and_wait
        filename=${line:1}
        echo "starting $filename" >&2
        coproc { sort >"$filename"; echo "Finished with $filename" >&2; }
        ;;
    *)
        printf '%s\n' "$line" >&${COPROC[1]}
        ;;
    esac
done
cleanup_and_wait

For prior versions of bash, a named pipe can be used instead:

cleanup_and_wait() {
    if [[ $child_pid ]] ; then
      exec 4<&-
      echo "waiting for $filename to finish" >&2
      wait $child_pid
    fi
}

# this is a bit racy; without a force option to mkfifo,
# however, the race is unavoidable
fifo_name=$(mktemp -u -t fifo.XXXXXX)
if ! mkfifo "$fifo_name" ; then
  echo "Someone else may have created our temporary FIFO before we did!" >&2
  echo "This can indicate an attempt to exploit a race condition as a" >&2
  echo "security vulnarability and should always be tested for." >&2
  exit 1
fi

# ensure that we clean up even on unexpected exits
trap 'rm -f "$fifo_name"' EXIT

while IFS= read -r line; do
    case $line in
    @*)
        cleanup_and_wait
        filename=${line:1}
        echo "starting $filename" >&2
        { sort >"$filename"; echo "finished with $filename" >&2; } <"$fifo_name" &
        child_pid=$!
        exec 4>"$fifo_name"
        ;;
    *)
        printf '%s\n' "$line" >&4
        ;;
    esac
done
cleanup_and_wait

A few notes:

• It's safer to use printf '%s\n' "$line" than echo "$line"; if a line contains only -e, for instance, some versions of echo will do nothing with it.
• Using an EXIT trap for cleanup ensures that an unexpected SIGTERM or other error won't leave the stale fifo sitting around.
• If your platform provides a way to create a FIFO with an unknown name in a single, atomic operation, use it; this would avoid the condition that requires us to always test whether the mkfifo is successful.

Answer 2

Easy, just pipe everything into cat.

#!/bin/bash
for i in `seq 5`; do
  {
  exec 3> >(sed -e "s/^/$i: /"; echo "$i-")
  echo foo >&3
  echo bar >&3
  exec 3<&-
  }|cat
done

Here's the output:

1: foo
1: bar
1-
2: foo
2: bar
2-
3: foo
3: bar
3-
4: foo
4: bar
4-
5: foo
5: bar
5-

Answer 3

mkfifo tmpfifo
for i in `seq 5`; do
  { sed -e "s/^/$i: /"; echo "$i-";} <tmpfifo &
  PID=$!
  exec 3> tmpfifo
  echo foo >&3
  echo bar >&3
  exec 3>&-
  wait $PID
done
rm tmpfifo

Answer 4

The "obvious" answer is to get rid of the process substitution.

for i in `seq 5`; do
    echo foo | sed -e "s/^/$i: /"; echo "$i-"
    echo bar | sed -e "s/^/$i: /"; echo "$i-"
done

So the question becomes, do you really need to structure your code using process substitution? The above is much simpler than trying to synchronize an asynchronous construct.

Answer 5

Another user asks the same question, and receives an exhaustive answer here.

Answer 6

It appears that you are required minimally to close the descriptor, and not sure whether it is required to wait pid. In the demo that follows, the sort does never fail when you don't wait $pidout but fail when you don't exec >&-

#!/bin/bash
seq 100000 |
(
while read;do
  # randomly fork an exec'd file descriptor
  if (( ! (pidout && RANDOM % 100) ));then
    # cleanup previous fork
    ((pidout)) && exec 1>&- && wait $pidout
    # fork &1 to >(subprocess)
    exec 1> >(
       # append my PID to each line
       # output to saved file descriptor
       sh -c 'exec sed "s/$/ $$/"' >&$fdout
    )
    pidout=$!
  fi
  # formatted line to exec'd file descriptor
  printf $'%6s\n' "$REPLY"
done
((pidout)) && exec 1>&- && wait $pidout
) {fdout}>&1 | # save $fdout file descriptor
# check output expected order
sort -k1,1n -c

However, a POSIX way that performs twice better, ensure the correct order, and is script miscible, is provided with few easily readable lines in awk. Used that in exact situations where you would have a process substitution associated to a while loop.

#!/bin/bash
seq 100000 |
awk 'BEGIN{srand()}
! (command && (rand() < 0.99)){
  if (command) close(command)
  command="sed \"s/$/ $$/\""   
}
{
  printf("%6s"ORS, $0)  | command
}' |
sort -k1,1n -c