How to make output of any shell command unbuffered?

Question

Is there a way to run shell commands without output buffering?

For example, hexdump file | ./my_script will only pass input from hexdump to my_script in buffered chunks, not line by line.

Actually I want to know a general solution how to make any command unbuffered?

more upvoted question http://unix.stackexchange.com/questions/25372 — Trevor Boyd Smith, Jun 10 '16 at 20:20
Perhaps see also https://stackoverflow.com/questions/7161821/how-to-grep-a-continuous-stream — tripleee, Feb 08 '21 at 05:51

score 172 · Answer 1 · answered Aug 28 '14 at 12:35

172

Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:

stdbuf -i0 -o0 -e0 command

answered Aug 28 '14 at 12:35

lambshaanxy

22,552
10
68
92

9

This worked better than `unbuffer` for me. `stdbuf` passed any signals (`SIGUSR2` in my case) I sent to it to the `command` (which is what I wanted to happen), while `unbuffer` didn't seem to want to. – ElDog Apr 11 '15 at 13:13
7

stdbuf uses one the aforementioned LD_PRELOAD tricks to do this and hence does not work with statically linked or setuid executables. See this question for a discussion: http://stackoverflow.com/questions/13644024/stdbuf-with-setuid-capabilities/18624182#18624182 – Nordic Mainframe Jul 04 '16 at 20:45
11

I found this very useful, but limited, as it does not run scripts implicitly. However, `stdbuf -o0 bash` runs a complete session with all my scripts and aliases available, and with no output buffering on any of the commands, which is exactly what I wanted. Buffering was of course restored on exit from this `bash` instance. I'm using Ubuntu 16.04. – AFH Jun 20 '17 at 12:13
2

I liked this so much, I put this first in my .bash_profile. `if [[ "$0" == "-bash" ]]; then exec /usr/bin/stdbuf -i0 -oL -eL /bin/bash -l; fi` – IcarusNM Aug 24 '19 at 20:19
2

Worked for me in an `exec` pattern for sending output to a logfile, with all output line buffered rather than buffering fully disabled: `exec > >(stdbuf -i0 -oL -eL awk '{print strftime("%Y-%m-%d %H:%M:%S"), $0 }' | stdbuf -i0 -oL -eL tee "$LOGFILE") 2>&1` – JinnKo Sep 06 '19 at 12:48
@NordicMainframe How can I verify `stdbuf` whether for a specific command or not,say `adb` – John Jun 01 '22 at 12:27
literally every, but openwrt. I bet other "embedded" busybox/toybox linuxes would also lack it. – Volodymyr Boiko Jun 22 '23 at 00:08

score 37 · Answer 2 · answered Aug 12 '10 at 08:50

37

The command unbuffer from the expect package disables the output buffering:
Ubuntu Manpage: unbuffer - unbuffer output

Example usage:

unbuffer hexdump file | ./my_script

answered Aug 12 '10 at 08:50

Siu Ching Pong -Asuka Kenji-

7,943
8
53
74

3

unbuffer seems to merge stdout and stderr of the command though... (!) – olejorgenb May 10 '16 at 19:57
3

@olejorgenb yes it merges stdout and stderr, that's by design: unbuffer uses expect, and expect is there to emulate the human in human-program interaction, and humans don't get to see whether an output is stdout or stderr. – toolforger Mar 31 '19 at 09:26

Nordic Mainframe · Accepted Answer · 2010-08-12T08:55:18.557

AFAIK, you can't do it without ugly hacks. Writing to a pipe (or reading from it) automatically turns on full buffering and there is nothing you can do about it :-(. "Line buffering" (which is what you want) is only used when reading/writing a terminal. The ugly hacks exactly do this: They connect a program to a pseudo-terminal, so that the other tools in the pipe read/write from that terminal in line buffering mode. The whole problem is described here:

http://www.pixelbeat.org/programming/stdio_buffering/

The page has also some suggestions (the aforementioned "ugly hacks") what to do, i.e. using unbuffer or pulling some tricks with LD_PRELOAD.

Thanks. I had never heard about pseudo terminals before. – bodacydo Aug 12 '10 at 16:07 — bodacydo, Aug 12 '10 at 16:07

score 23 · Answer 4 · edited Mar 11 '16 at 15:56

23

You could also use the script command to make the output of hexdump line-buffered (hexdump will be run in a pseudo terminal which tricks hexdump into thinking its writing its stdout to a terminal, and not to a pipe).

# cf. http://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe/
stty -echo -onlcr
script -q /dev/null hexdump file | ./my_script         # FreeBSD, Mac OS X
script -q -c "hexdump file" /dev/null | ./my_script    # Linux
stty echo onlcr

edited Mar 11 '16 at 15:56

Community

1
1

answered May 07 '13 at 15:00

vroc

231
2
2

I used the -f parameter to flush, not sure if necessary but it worked. – Gringo Suave Oct 24 '13 at 01:43
In my case, `stdbuf` failed, but script works like a charm. The program buffers the output when it is being piped to `tee`, but not when running in terminal. So, script works! Thanks! – TerrenceSun May 18 '16 at 12:11
Very clever solution, and it works!! Thank you. :smile: – xmnboy Mar 01 '17 at 01:26
Only one problem with that: per `script`'s `man` page as referenced by [this](https://unix.stackexchange.com/a/86902/86927), "Certain interactive commands, such as `vi(1)`, create garbage in the typescript file. Script works best with commands that do not manipulate the screen, the results are meant to emulate a hardcopy terminal." – RandomDSdevel Nov 02 '17 at 00:01

score 1 · Answer 5 · answered May 17 '21 at 06:19

1

One should use grep or egrep "--line-buffered" options to solve this. no other tools needed.

answered May 17 '21 at 06:19

Binghoo Dang

19
1

How to make output of any shell command unbuffered?

5 Answers5

Linked

Related