10

When designing a chain of commands to perform a certain task, i ran into the problem that anonymous pipes do not behave like expected. As the original command that i am running is too complex to explain here, i've created an example that shows the problem (i know that all these commands are doing basically nothing). Also, i am using pv to show whether the data is actually copied from input to output.

cat /dev/zero | pv > /dev/null

This works as expected. (copy data from /dev/zero to /dev/null)

cat /dev/zero | tee /dev/null | pv > /dev/null

This also works as expected (duplicate the data and send both copies to /dev/null)

cat /dev/zero | tee >(pv -c > /dev/null) | pv -c > /dev/null

This command only partially works. While the copy from STDIN to STDOUT still works, (one pv will show progress for a short time), the whole command gets stalled by the anonymous pipe, which does not receive anything and thus tee stalls as one of the outputs cannot be written to (I checked this by letting it write to files instead of /dev/null).

If anyone has an Idea why this does not work (as expected ?) in bash, i'd be glad for the help.

PS: If I use zsh instead of bash, the command runs as expected. Unfortunately, the system this needs to run on has no zsh and there is no way for me to get zsh on that system deployed.

HDJEMAI
  • 9,436
  • 46
  • 67
  • 93
Fabraxias
  • 221
  • 3
  • 7
  • Running it directly via `bash -c` or in a subshell (i.e.: `(cat /dev/zero | tee >(pv -c > /dev/null) | pv -c > /dev/null)`) seems to work... running it in an interactive shell it hangs like you mentioned. I have no idea why. – FatalError Nov 17 '15 at 09:52
  • 1
    Does this happen with `pv` only? Because when I substitute the first `pv` with `cat`, it works for me. Maybe `pv` just doesn't work with process substitution magic? – Vasily G Nov 17 '15 at 11:44
  • As far as i know, cat does not read from STDIN. By replacing the first pv with cat, the read from the anonymous pipe is effectively ignoring the anonymous redirect. I don't think that that is the desired effect. However, replacing pv with something else (like awk '{print $0}' or perl -ne 'print $_' has the same effect (both commands effectively copy STDIN to STDOUT) – Fabraxias Nov 18 '15 at 12:18
  • 1
    @Fabraxias : cat does read from std-in. Try `echo "abc" | cat` or if you have an old-line unixen, you might need `echo "abc" | cat -` (the `-` says read from std-in). Good luck to all. – shellter Nov 18 '15 at 19:54
  • @shellter - Just a comment on your comment -- even in newer unicies, the `-` for stdin can be pretty handy. Consider: `yadda | cat header.txt - footer.txt` – ghoti Nov 19 '15 at 12:54

1 Answers1

1

Whe you use <( ... ) for process substitution, the process running inside does not have a controlling terminal. But pv always shows its results to the terminal; if there isn't any, it gets stopped.

If you execute your code and, while it is running, do a ps axf, you will see something like this:

23412 pts/16   S      0:00  \_ bash
24255 pts/16   S+     0:00      \_ cat /dev/zero
24256 pts/16   S+     0:00      \_ tee /dev/fd/63
24258 pts/16   S      0:00      |   \_ bash
24259 pts/16   T      0:00      |       \_ pv -c
24257 pts/16   S+     0:00      \_ pv -c

...which tells you that the pv -c executed inside the process substitution (the one below the second bash) is in T state, stopped. It is waiting to have a controlling terminal in order to run. It does not have any, so it will stop forever, and bash eventually stops sending data to that pipe.

Juan Cespedes
  • 1,299
  • 12
  • 27
  • mhm - ok, I understand that for <() - that redirecting STDIN of the subshell to the outshell by the an anonymous pipe. So, doing a "cat <(gunzip -c some.gz)" would actually work (yes, it another useless command and a very roundabout way for zcat). But, as far as i understand it, >() does exactly the opposite - redirect the output from the outer shell to the inner shell via an anonymous pipe. In other words, shouldn't the /dev/fd/63 be the automatic input for the stopped pv ? – Fabraxias Nov 19 '15 at 22:28