25

I've got a script like:

#!/bin/bash
exec /usr/bin/some_binary > /tmp/my.log 2>&1

Problem is that some_binary sends all of its logging to stdout, and buffering makes it so that I only see output in chunks of a few lines. This is annoying when something gets stuck and I need to see what the last line says.

Is there any way to make stdout unbuffered before I do the exec that will affect some_binary so it has more useful logging?

(The wrapper script is only setting a few environment variables before the exec, so a solution in perl or python would also be feasible.)

Nakilon
  • 34,866
  • 14
  • 107
  • 142
bstpierre
  • 30,042
  • 15
  • 70
  • 103
  • 6
    Dennis's answer is *exactly* right. Here's why. Programs on Unix that use C stdio for output will buffer when the output is not going to a terminal (boosts performance a lot) unless told not to with `setvbuf`; hardly anyone bothers with that. The `expect` system (of which `unbuffer` is a trivial application) uses black magic with Unix pseudo-terminal devices (ptys) to run programs with a terminal that isn't a real terminal so that the wrapped program doesn't buffer its output. It's clever, but be warned: most systems only have a limited number of ptys; a naturally unbuffered program is better. – Donal Fellows Jul 26 '10 at 08:32
  • BTW, this doesn't work so well on Windows. That's because expect on that platform uses debugging magic instead of terminal magic (because this is an area where Windows is *very* different). – Donal Fellows Jul 26 '10 at 08:34
  • Thanks. I decided to opt for modifying `some_binary` to do the right thing instead of relying on the black magic. `expect` is great, but I don't want it here... – bstpierre Jul 26 '10 at 14:09

6 Answers6

35

GNU coreutils-8.5 also has the stdbuf command to modify I/O stream buffering:

http://www.pixelbeat.org/programming/stdio_buffering/

So, in your example case, simply invoke:

stdbuf -oL /usr/bin/some_binary > /tmp/my.log 2>&1

This will allow text to appear immediately line-by-line (once a line is completed with the end-of-line "\n" character in C). If you really want immediate output, use -o0 instead.

This way could be more desirable if you do not want to introduce dependency to expect via unbuffer command. The unbuffer way, on the other hand, is needed if you have to fool some_binary into thinking that it is facing a real tty standard output.

bschlueter
  • 3,817
  • 1
  • 30
  • 48
zaga
  • 351
  • 1
  • 2
  • 2
14

You might find that the unbuffer script that comes with expect may help.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • Does not work with the `echo -n .` or `prinff "."` which I need in my case to show a progressbar in bash – Davide Dec 31 '13 at 17:27
  • See also: http://stackoverflow.com/questions/1401002/trick-an-application-into-thinking-its-stdin-is-interactive-not-a-pipe – Mihai Danila Nov 25 '14 at 21:01
11

Some command line programs have an option to modify their stdout stream buffering behaviour. So that's the way to go if the C source is available ...

# two command options ...
man file | less -p '--no-buffer'
man grep | less -p '--line-buffered'

# ... and their respective source code

# from: http://www.opensource.apple.com/source/file/file-6.2.1/file/src/file.c
if(nobuffer)
   (void) fflush(stdout);

# from: http://www.opensource.apple.com/source/grep/grep-28/grep/src/grep.c
if (line_buffered)
   fflush (stdout);

As an alternative to using expect's unbuffer script or modifying the program's source code, you may also try to use script(1) to avoid stdout hiccups caused by a pipe:

See: Trick an application into thinking its stdin is interactive, not a pipe

# Linux
script -c "[executable string]" /dev/null

# FreeBSD, Mac OS X
script -q /dev/null "[executable string]"
Community
  • 1
  • 1
trevor
  • 111
  • 2
  • 1
    On Mac OS X, because "script" may change return code from "\n" to "\r\n", I needed to run like this: `script -q /dev/null commands... | perl -pe 's/\n/\r\n/g'` – Tsuneo Yoshioka Nov 27 '12 at 15:54
3

I scoured the internets for an answer, and none of this worked for uniq which is too stubborn to buffer everything except for stdbuf:

{piped_command_here} | stdbuf -oL uniq | {more_piped_command_here}

Dexter Legaspi
  • 3,192
  • 1
  • 35
  • 26
2

GNU Coreutils-8 includes a program called stdbuf which essentially does the LD_PRELOAD trick. It works on Linux and reportedly works on BSD systems.

Otheus
  • 566
  • 5
  • 7
2

An environment variable can set the terminal IO mode to unbuffered.

export NSUnbufferedIO=YES

This will set the terminal unbuffered for both C and Ojective-C terminal output commands.

EKM
  • 21
  • 1