4

System: Linux 4.13.0-43-generic #48~16.04.1-Ubuntu BASH_VERSION='4.3.48(1)-release'

The command:

while sleep 5
do
  date +%T
done | awk -F: '{print $3}'

Should print the 3rd field (seconds) of the "date" output, one line every 5 seconds. Problem: awk reads from the pipe, and processes its input, only when the pipe's buffer is full. i.e. when more than 4K of input is generated.

When awk is replaced by cat, a line is printed every 5 seconds as expected.

This code snippet is simplified from a shell script which had worked ok on other systems, so there must be something about bash, awk and their configuration in this system.

In short, is there a way to convince awk to behave like cat when reading from a pipe?

@Ed Morton : I did try to add fflush() after each print, but it does not work -- that's what showed that the problem is with awk's input, not output. I also tried to add calls to system("date"), which showed that indeed awk gets all input lines at once, not immediately when they are produced.

For those who asked:

$ awk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan

compiled limits:
max NF             32767
sprintf buffer      2040
fedorqui
  • 275,237
  • 103
  • 548
  • 598
Amos Shapir
  • 91
  • 7
  • The awk version is more pertinent than the bash version here. That said, I can't reproduce this at all -- and indeed, it **shouldn't** be reproducible, as `date` necessarily flushes its output before it can exit. – Charles Duffy Jun 06 '18 at 14:59
  • Could you perhaps provide a Dockerfile where the issue reproduces as-described? I've tried two separate environments, and the code sample in this question behaves properly. – Charles Duffy Jun 06 '18 at 15:03
  • The problem may be related to the surrounding code which you trimmed away, e.g. if you're piping the awk output to another command. In any case, try adding a call to fflush() after the print to see if that helps - `{print $3; fflush()}`. See https://www.gnu.org/software/gawk/manual/gawk.html#I_002fO-Functions for details – Ed Morton Jun 06 '18 at 15:21
  • 2
    When you simplify code, please make sure to test it *after* you simplify. This helps ensure you don't accidentally remove the relevant parts. – that other guy Jun 06 '18 at 15:50
  • Hi, it does work for me, I've also tried `while sleep 5; do date +%T | cut -d: -f3;done` – David Peltier Jun 06 '18 at 16:46
  • See also https://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered – tripleee Feb 08 '21 at 05:37

1 Answers1

5

While trying to find out how to make awk print its version, I discovered that it is really mawk, and that it has the following flag:

 -W interactive -- sets unbuffered writes to stdout and line buffered reads from stdin.
                   Records from stdin are lines regardless of the value of RS.

This seems to solve the problem!

Thanks to all repliers.

Amos Shapir
  • 91
  • 7