0

I have a bash pipeline. I would like to stop too much work being by the earlier workers before the later workers have processed it. Is there a way to decrease the amount data that can be stored in a pipe and I can I specify this limit in lines rather than bytes?

I'm aware that the "correct" way of doing this would be to use a message passing system -but that seems like a bunch of overhead.

Att Righ
  • 1,439
  • 1
  • 16
  • 29
  • 2
    That's up to the individual programs to configure for themselves, IIRC. They'll be given data as often as they ask for it – Alexander Mar 22 '22 at 14:57
  • Once the buffer fills up, the pipeline should block, right? Are you seeing a problem, or just thinking ahead to one that may not exist? If you are really concerned, does https://stackoverflow.com/questions/51636391/how-to-set-pipe-size-from-shell help? – Paul Hodges Mar 22 '22 at 16:12

1 Answers1

1

Yes, you can with the pv utility and the -l switch. For example:

$dmesg |pv -qL 1 -l

will display kernel info one line per second.

$cat my_file.txt | pv -L 5 -l | some_program

will send the contents of my my_file.txt through pv and pv will pass it to some_program at the selected rate. Without the -q switch you get a progress bar corresponding the data being send through the pipe. Without -l the parameter is taken as bytes per second. More details on the manual man pv

Install it through your package manager pv.

Ubuntu: apt install pv

OpenSuse: zypper in pv

etc.

RedComet
  • 1,192
  • 5
  • 11