3

I have a script that wants to run several programs / pipelines over a very large file. Example:

grep "ABC" file > file.filt
md5sum file > file.md5

The kernel will try to cache the file in RAM, so if it is read again soon it may be a copy from RAM. However the files are large, and the programs run at wildly different speeds, so this is unlikely to be effective. To minimise IO usage I want to read the file once.

I know of 2 ways to duplicate the data using tee and moreutils' pee:

<file tee >(md5sum > file.md5) | grep "ABC" > file.filt
<file pee 'md5sum > file.md5' 'grep "ABC" > file.filt'

Is there another 'best' way? Which method will make the fewest copies? Does it make a difference which program is >() or |-ed to? Will any of these approaches attempt to buffer data in RAM if one program is too slow? How do they scale to many reader programs?

Evan Benn
  • 1,571
  • 2
  • 14
  • 20
  • 1
    In your last two attempts, your are missing re-direction to the `file.filt`, it should be `> file.filt` right? Also you are reading the `file` once, and your logic looks alright! What are trying to improve? Is it still slower? – Inian Aug 14 '18 at 08:42

2 Answers2

1

tee (command) opens each file using fopen, but sets _IONBF (unbuffered) on each. It reads from stdin, and fwrites to each FILE*.

pee (command) popens each command, sets each to unbuffered, reads from stdin, and fwrites to each FILE*. popen uses pipe(2), which has a capacity of 65536 bytes. Writes to a full buffer will block. pee also uses /bin/sh to interpret the command, but I think that will not add any buffering/copying.

mkfifo (command) uses mkfifo (libc), which use pipes underneath, opening the file/pipe blocks until the other end is opened.

bash <>() syntax (subst.c:5712) uses either pipe or mkfifo. pipe if /dev/fds are supported. It does not use the c fopen calls so does not set the buffering.

So all three variants (pee, tee >(), mkfifo ...) should end up with identical behaviour, reading from stdin and writing to pipes without buffering. The data is duplicated at each read (from kernel to user), and then again at each write (user back to kernel), I think tees fwrites will not cause an extra layer of copying (as there is no buffer). Memory usage could increase to a maximum of 65536 * num_readers + 1 * read_size (if no one is reading). tee writes to stdout first, then each file/pipe in order.

Given this pee just works around other shells (fish!) lack of >() operator equivalent, there seems to be no need for it with bash. I prefer tee when you have bash, but pee is nice when you don't. The bash <() is not replaced by pee of course. Manually mkfifoing and redirecting is tricky and unlikely to deal with errors nicely.

pee could probably be changed by implementing using the tee library function (instead of fwrite). I think this would cause the input to be read at the speed of the fastest reader, and potentially fill up the kernel buffers.

Evan Benn
  • 1,571
  • 2
  • 14
  • 20
0

AFAIK, there is no "best way" to achieve this. But I can give you another approach, more verbose, not a one liner, but maybe clearer because each command is written in its own. Use named pipes:

mkfifo tmp1 tmp2
tee tmp1 > tmp2 < file &
cat tmp1 | md5sum > file.md5 &
cat tmp2 | grep "ABC" > file.filt &
wait
rm tmp1 tmp2
  1. Create as many name pipes as commands to be run.
  2. tee to the named pipes the input file (tee outputs its input in the standard output, so the last name pipe must be a redirection), let it run in background.
  3. Use the different named pipes as input to the different commands to run. Let them run in the background.
  4. Finally, wait for the jobs to finish and remove the temporary named pipes.

The drawback of this approach, when the programs have gread variability in theirs speeds is that all of them will read the files at the same pace (limit is the buffer size, once full for one of the pipes, the others will have to wait too), so if one of them is resource-hungry (like memory hungry), the resources will be used for the whole lifespan of all the processes.

Poshi
  • 5,332
  • 3
  • 15
  • 32
  • I disagree that this is clearer, it removes the not well known bash >() syntax, but requires you use the primitives to do the same thing. It does not die as gracefully. – Evan Benn Aug 14 '18 at 10:52
  • You are right. It does the same thing, I already said it is another approach to the same. It does not die gracefully on a block. It is just another way to do the requested task. And one that avoid having lines of thousands characters. If you have to run 1000 processes with the same input, you will easily get to the "line too long" error in the one-liner approach. – Poshi Aug 14 '18 at 11:55
  • I think the line length limit is for argv + env right? So `pee` will certainly hit that limit. But tee with >() is many separate command lines, and your approach still uses `tee a b c`, so would hit the same limit no? – Evan Benn Aug 15 '18 at 23:23
  • On one side, len("tee a") – Poshi Aug 16 '18 at 08:31
  • yes the cascaded tees makes sense. But on the other point, "tee >(abc)" is interpreted by bash, not the loader, and there is no length limit in bash (that I can find). bash turns "tee >(abc)" into 2 commands "abc" and "tee /dev/fdx". – Evan Benn Aug 16 '18 at 11:45
  • There is no length limit in bash, but in the OS: https://stackoverflow.com/questions/19354870/bash-command-line-and-input-limit – Poshi Aug 16 '18 at 12:34