sed and tee when working together in pipes no longer behaves like stream

Question

Anyone can help explaining why sed and tee when used separately seems to behave like stream but not when combined?

You can see how stdout is progressively being streamed to stdout.

But when used together you can see stdout only at the end of the execution of the main process in the pipe.

Examples:

Note:

perl -pe "system 'sleep .03'"

is just for throttling stdout to see final stdout progressively building up

and

sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

is for stripping out colours


printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

^^^ in above example you can clearly see that stdout is building progressively


printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | tee color.txt

^^^ using just tee is fine too


printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g' | tee color.txt

^^^ but when combined there is clearly buffering in play.

What's causing that? If anyone can explain please?

Edited:

when sed with tee swapped it is not buffering anymore, but obviously is not what I want


printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'"  | tee color.txt | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

All of that behaves exactly the same on:

mac with zsh
mac with bash
docker run -it ubuntu bash

docker run -it ubuntu sh

  $ sw_vers

  ProductName:    macOS
  ProductVersion: 11.6
  BuildVersion:   20G165

  $ /bin/zsh --version

  zsh 5.8 (x86_64-apple-darwin20.0)

  $ /bin/bash --version

  GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
  Copyright (C) 2007 Free Software Foundation, Inc.

This is [BashFAQ #9](https://mywiki.wooledge.org/BashFAQ/009) — Charles Duffy, Apr 07 '22 at 14:09
(and neither bash nor zsh has anything to do with the behavior in question; it's the standard C library that decides what kind of buffering to do by default based on whether one is going to a TTY) — Charles Duffy, Apr 07 '22 at 14:11

score 1 · Accepted Answer · answered Apr 07 '22 at 13:58

1

stdout is usually line buffered only if connected to a terminal. Connecting it to a pipe causes full buffering.

Although some commands, including tee, are guaranteed not to buffer at all (hence the last example of tee | sed).

You can use stdbuf -oL sed 's...g' or unbuffer sed 's...g' to get line buffering. stdbuf is part of GNU coreutils so should be available on Linux, It's also available by default on FreeBSD.

This article from 2006 is old, but provides some good detail. The author is a GNU coreutils maintainer.

http://www.pixelbeat.org/programming/stdio_buffering/

answered Apr 07 '22 at 13:58

dan

4,846
6
15

Makes sense, thanks for answer. My case was fixed by adding unbuffer -p (unbuffer according to manpage should have -p when used with pipes) before sed. Indeed tee seems to always "not buffer". – stopsopa Apr 08 '22 at 11:05
1

More details on how tee should work are in the POSIX spec, specifically under "Rationale". https://pubs.opengroup.org/onlinepubs/009696699/utilities/tee.html – dan Apr 09 '22 at 03:26

sed and tee when working together in pipes no longer behaves like stream

1 Answers1