8

When I issued two equivalent commands in Bash I got different output (from "wc -l" command), see below:

root@devel:~# ls /usr/bin -lha | tee >(wc -l) >(head) > /dev/null
total 76M
drwxr-xr-x  2 root root      20K Nov 11 18:58 .
drwxr-xr-x 10 root root     4.0K Oct  8 15:31 ..
-rwxr-xr-x  1 root root      51K Feb 22  2017 [
-rwxr-xr-x  1 root root       96 Jan 19  2017 2to3-3.5
-rwxr-xr-x  1 root root      23K Mar 22  2017 addpart
lrwxrwxrwx  1 root root       26 May 10  2017 addr2line -> x86_64-linux-gnu-    addr2line
lrwxrwxrwx  1 root root        6 Dec 13  2016 apropos -> whatis
-rwxr-xr-x  1 root root      15K Sep 13 19:47 apt
-rwxr-xr-x  1 root root      79K Sep 13 19:47 apt-cache
137
root@devel:~# ls /usr/bin -lha | wc -l
648

what am I missing?

it's strange, but when I call it this way it gets even stranger output:

root@devel:~# ls /usr/bin -lha | tee >(wc) >(wc) > /dev/null
648    6121   39179
648    6121   39179
root@devel:~# ls /usr/bin -lha | tee >(wc) >(wc) > /dev/null
648    6121   39179
648    6121   39179
root@devel:~# ls /usr/bin -lha | tee >(wc) >(wc -l) > /dev/null
648
root@devel:~#     648    6121   39179

seems like commands running asynchronously and ends in different time... or what it can be?

urusai_na
  • 393
  • 4
  • 11
  • 1
    Yes, process substitution is run asynchronously, see [the manual](https://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html). – Benjamin W. Nov 12 '17 at 20:01
  • 1
    ok, but how to fix it, how to make program wait until real output ends? or there aren't any solutions here? – urusai_na Nov 12 '17 at 20:07
  • What exactly do you want to achieve? Get both the output of `ls | head` and `ls | wc -l`? – Benjamin W. Nov 12 '17 at 20:24
  • yes, but "wc -l" gets only 137 lines instead of 648, why? it seems not related to asynchronicity because in second variant I always get 648 lines – urusai_na Nov 12 '17 at 20:29
  • actually, I want process same output with multiple set of commands, but it seems not that easy and straightforward as I expected it to be and works differently with different commands or I do something wrong way... – urusai_na Nov 12 '17 at 20:32
  • I'm stumped, to be honest. The second block of code makes sense, but I don't understand what's happening in the first one. – Benjamin W. Nov 12 '17 at 21:11
  • Yes @dave_thompson_085 your find is correct. OP what do you want to do? There has got to be another way to accomplish what you want, without running into that condition. – Nic3500 Nov 12 '17 at 23:39

1 Answers1

8

Simple answer:

how to fix:

ls /usr/bin -lha | tee --output-error=exit-nopipe >(wc -l) >(head) > /dev/null

Details:

The command head only prints the head of input, so it can finish its job as long as it gets enough input, then exits without waiting for all input.

So let's replace command head with the simple "head".

ls /usr/bin -lha | tee >(wc -l) >(read l; echo $l) > /dev/null

The simple "head" will read only one line, then exit, which causes that the pipe file gets closed immediately before tee finishes transferring all data to it.

So no doubt, you'll get same result with the simple "head". wc still prints wrong number.

The root reason of your issue, I think you can conclude yourself, is that one of the output pipes of tee is closed earlier, tee hits a write error, and then stops writing to other output files.

After understanding the root reason, I think it would be very easy for you to understand the following section in man page.

MODE determines behavior with write errors on the outputs:
   'warn' diagnose errors writing to any output

   'warn-nopipe'
          diagnose errors writing to any output not a pipe

   'exit' exit on error writing to any output

   'exit-nopipe'
          exit on error writing to any output not a pipe

   The  default MODE for the -p option is 'warn-nopipe'.  The default operation
   when --output-error is not specified, is to exit immediately on error writing to
   a pipe, and diagnose errors writing to non pipe outputs.

Some extra words

Actually if you replace >(wc -l) with a regular file in your problematic command line, you will find the file size will always be 16384 or 20480 or 32768 or 36864 or 28672 or ..., all of which are the multiple of 4096. (The writing to the regular file is incomplete because tee aborts earlier. If the writing was complete, the file size would be any value.)

4096 is the value of PIPE_BUF for most UNIX-like system. If you know what PIPE_BUF is, you will easily understand why the file size is always the multiple of 4096.

Bruce
  • 523
  • 5
  • 8
  • Explanation of the command [here](https://explainshell.com/explain?cmd=ls+%2Fusr%2Fbin+-lha+%7C+tee+--output-error%3Dexit-nopipe+%3E%28wc+-l%29+%3E%28head%29+%3E+%2Fdev%2Fnull) – Sebastian Breit Nov 17 '17 at 02:43