169

Usually, stdout is line-buffered. In other words, as long as your printf argument ends with a newline, you can expect the line to be printed instantly. This does not appear to hold when using a pipe to redirect to tee.

I have a C++ program, a, that outputs strings, always \n-terminated, to stdout.

When it is run by itself (./a), everything prints correctly and at the right time, as expected. However, if I pipe it to tee (./a | tee output.txt), it doesn't print anything until it quits, which defeats the purpose of using tee.

I know that I could fix it by adding a fflush(stdout) after each printing operation in the C++ program. But is there a cleaner, easier way? Is there a command I can run, for example, that would force stdout to be line-buffered, even when using a pipe?

oguz ismail
  • 1
  • 16
  • 47
  • 69
houbysoft
  • 32,532
  • 24
  • 103
  • 156

7 Answers7

167

you can try stdbuf

$ stdbuf --output=L ./a | tee output.txt

(big) part of the man page:

  -i, --input=MODE   adjust standard input stream buffering
  -o, --output=MODE  adjust standard output stream buffering
  -e, --error=MODE   adjust standard error stream buffering

If MODE is 'L' the corresponding stream will be line buffered.
This option is invalid with standard input.

If MODE is '0' the corresponding stream will be unbuffered.

Otherwise MODE is a number which may be followed by one of the following:
KB 1000, K 1024, MB 1000*1000, M 1024*1024, and so on for G, T, P, E, Z, Y.
In this case the corresponding stream will be fully buffered with the buffer
size set to MODE bytes.

keep this in mind, though:

NOTE: If COMMAND adjusts the buffering of its standard streams ('tee' does
for e.g.) then that will override corresponding settings changed by 'stdbuf'.
Also some filters (like 'dd' and 'cat' etc.) dont use streams for I/O,
and are thus unaffected by 'stdbuf' settings.

you are not running stdbuf on tee, you're running it on a, so this shouldn't affect you, unless you set the buffering of a's streams in a's source.

Also, stdbuf is not POSIX, but part of GNU-coreutils.

Flow
  • 23,572
  • 15
  • 99
  • 156
c00kiemon5ter
  • 16,994
  • 7
  • 46
  • 48
  • 3
    Thanks, but this does not seem to be available on OS X (the question is tagged osx-lion). – houbysoft Jul 05 '12 at 02:58
  • 3
    @houbysoft - I am pretty sure GNU tools can be installed on OS X – jordanm Jul 05 '12 at 03:00
  • 2
    @jordanm: perhaps, but installing the entire GNU tools kind of seems like overkill for this... – houbysoft Jul 05 '12 at 03:06
  • 3
    Upvoted this answer because `stdbuf` is already available on the Centos Linux distributions we're using, and `unbuffer` isn't. Thanks! – Huw Walters Feb 06 '19 at 09:05
  • 29
    For python script stdbuf will not work, but you can use `-u` to disable buffering on python's side: `python3 -u a.py | tee output.txt` – Honza Jan 02 '20 at 13:56
  • 1
    `PYTHONUNBUFFERED=1 python3 a.py | tee output.txt` is another alterative – James McGuigan Jun 09 '20 at 21:59
  • Didn't work for me on GitHub Actions. For something as simple as `apk add coreutils && stdbuf -o0 -e0 -i0 sh -xc 'set && whoami && pwd && ls && exit 1'`. With [`unbuffer`](https://stackoverflow.com/a/11337310/52499) it works. – x-yuri Aug 09 '20 at 13:50
  • This is the code where tee adjusts the buffering https://github.com/coreutils/coreutils/blob/master/src/tee.c#L208 – hailinzeng May 16 '22 at 06:35
86

Try unbuffer (man page) which is part of the expect package. You may already have it on your system.

In your case you would use it like this:

unbuffer ./a | tee output.txt

The -p option is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • Thanks, this worked, although I had to compile `expect` myself as `unbuffer` doesn't seem to be included by default in OS X. – houbysoft Jul 05 '12 at 03:11
  • @houbysoft: I'm glad it worked for you. `unbuffer` is only a small script so you shouldn't have needed to recompile the whole package. – Dennis Williamson Jul 05 '12 at 03:40
  • Yeah, probably not, but `./configure && make` took about 10 seconds and then I just moved `unbuffer` to `/usr/local/bin` :) – houbysoft Jul 05 '12 at 03:45
  • 3
    I got it installed on my mac (10.8.5) via brew: brew install expect --with-brewed-tk – Nils Sep 18 '13 at 19:07
  • 2
    FWIW, because unbuffer is somewhat confusing, the relevant structure is `unbuffer {commands with pipes/tee}`. – Fake Name Nov 20 '15 at 04:59
  • I had to install it this way: `brew install homebrew/dupes/expect` – nhed May 11 '16 at 19:49
  • This has issues for me when the program requires input on the command line: it doesn't print the prompt (unless I press enter, and then it's too late as Enter is accepted as the input to the prompt). `stdbuf` however worked. – DBedrenko Oct 21 '18 at 10:46
  • 1
    I think it should rather be "unbuffer ./a | tee output.txt" - it is not tee that needs unbuffering. That worked for me, at least, for a similar problem. – Hugo van der Sanden Aug 18 '19 at 03:41
  • on macOS, `brew install expect` works fine these days. (and options like `--with-brewed-tk` have been removed from all core formulas) – Chris Jun 02 '20 at 09:35
  • 1
    I think the right command if you read the man page is `unbuffer ./a | tee output.txt`. That's what worked for me under RPi with bash and tmux. – berezovskyi Aug 07 '20 at 15:01
  • I have corrected the command which was added in an edit by another user. – Dennis Williamson Jun 28 '22 at 15:43
37

You can use setlinebuf from stdio.h.

setlinebuf(stdout);

This should change the buffering to "line buffered".

If you need more flexibility you can use setvbuf.

Denys Rtveliashvili
  • 479
  • 4
  • 3
  • 10
    I wonder why this solution has so few upvotes. This is the only solution not imposing a burden on the caller. – oxygene May 25 '16 at 06:29
  • 5
    Note that this isn't standard C (or even POSIX). It's probably better to use `setvbuf(stdout, NULL, _IOLBF, 0)`, which is exactly equivalent. – rvighne Oct 11 '19 at 07:13
  • This fixed my issue on OS X Catalina with a C++ program which was printf()ing and I was piping to tee but was only seeing the output when the program had finished. – jbaxter Jan 15 '20 at 06:41
34

You may also try to execute your command in a pseudo-terminal using the script command (which should enforce line-buffered output to the pipe)!

script -q /dev/null ./a | tee output.txt     # Mac OS X, FreeBSD
script -c "./a" /dev/null | tee output.txt   # Linux

Be aware the script command does not propagate back the exit status of the wrapped command.

bk-se
  • 540
  • 5
  • 7
jon
  • 341
  • 2
  • 2
  • 5
    `script -t 1 /path/to/outputfile.txt ./a` worked great for my use case. It live streams all output to `outputfile.txt` while also printing it to your shell's stdout. Didn't need to use `tee` – Peter Berg Aug 14 '15 at 13:25
  • `script` from util-linux and BSD both offer the `-e` option to return the exit status of the executed command. – Socowi Aug 14 '21 at 13:38
8

The unbuffer command from the expect package at the @Paused until further notice answer did not worked for me the way it was presented.

Instead of using:

./a | unbuffer -p tee output.txt

I had to use:

unbuffer -p ./a | tee output.txt

(-p is for pipeline mode where unbuffer reads from stdin and passes it to the command in the rest of the arguments)

The expect package can be installed on:

  1. MSYS2 with pacman -S expect
  2. Mac OS with brew install expect

Update

I recently had buffering problems with python inside a shell script (when trying to append timestamp to its output). The fix was to pass -u flag to python this way:

  1. run.sh with python -u script.py
  2. unbuffer -p /bin/bash run.sh 2>&1 | tee /dev/tty | ts '[%Y-%m-%d %H:%M:%S]' >> somefile.txt
  3. This command will put a timestamp on the output and send it to a file and stdout at the same time.
  4. The ts program (timestamp) can be installed with the moreutils package.

Update 2

Recently, also had problems with grep buffering the output, when I used the argument grep --line-buffered on grep to it stop buffering the output.

Evandro Coan
  • 8,560
  • 11
  • 83
  • 144
  • The same happened to me. In this way, it worked. – Carlos Pinzón Jul 23 '21 at 11:20
  • You made my day with the python thing! BTW. you can also use env variable like so with the same effect: export PYTHONUNBUFFERED=1 – Airstriker May 24 '22 at 11:42
  • If you need logging in the bash script this is a really nice setup: # Send all output to one file and all output to the screen LOGFILE=some_file.log set -x #prefix each line with a timestamp and print it out (without buffering) exec > >(ts "[%F %H:%M:%S]" | unbuffer -p tee ${LOGFILE}) 2>&1 # force python stdout and stderr streams to be unbuffered; export PYTHONUNBUFFERED=1 – Airstriker May 24 '22 at 12:04
3

If you use the C++ stream classes instead, every std::endl is an implicit flush. Using C-style printing, I think the method you suggested (fflush()) is the only way.

Kevin Grant
  • 5,363
  • 1
  • 21
  • 24
  • 4
    Unfortunately, this is not true. You can observe the same behavior with c++ std::cout even when using std::endl or std::flush. The buffering happens on-top and the easiest solution in Linux seems to be setlinebuf(stdout); as the very first line in main() when you are the author of the program and using the other above solutions when not being able to change the source code. – oxygene May 25 '16 at 06:25
  • 2
    @oxygene This is not true. I tried it and endl does flush the buffer when piping to tee (unlike with printf). Code: `#include #include int main(void) { std::cout << "1" << std::endl; sleep(1); std::cout << "2" << std::endl; }`. endl always flushes the buffer as defined here: http://en.cppreference.com/w/cpp/io/manip/endl – Curtis Yallop Jul 07 '17 at 16:39
1

The best answer IMO is grep's --line-buffer option as stated here:

https://unix.stackexchange.com/a/53445/40003

Marcello Romani
  • 2,967
  • 31
  • 40
  • I came here to add `grep --line-buffered` as a simple alternative... however, I am noticing with multiple parallel processes, I'm still sometimes seeing lines that get joined (but, at least, it's complete lines getting joined at once, instead of lines interleaving at random columns...) – Ajax Jan 17 '23 at 23:34