python stdout flush and tee

Question

The following code ends with broken pipe when piped into tee, but behave correctly when not piped :

#!/usr/bin/python
import sys
def testfun():
    while 1:
        try :
            s = sys.stdin.readline()
        except(KeyboardInterrupt) :
            print('Ctrl-C pressed')
            sys.stdout.flush()
            return
        print s

if __name__ == "__main__":
    testfun()
    sys.exit()

Expected output :

./bug.py 
Ctrl-C pressed

What is observed when piped into tee is either a broken pipe or no output at all, ie nothing on tee stdout, and nothing in bug.log :

./bug.py | tee bug.log
Traceback (most recent call last):
  File "./bug.py", line 14, in <module>
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe

What can be the reason for this ?

score 9 · Answer 1 · answered Aug 31 '11 at 00:52

9

Nope, hitting Ctrl-C does NOT terminate both processes. It terminates the tee process only, the end of the tee process close the pipe between your script and tee, and hence your script dies with the broken pipe message.

To fix that, tee has an option to pass the Ctrl-C to its previous process in the pipe: -i

try: man tee

./bug.py
^CCtrl-C pressed
./bug.py | tee log
^CTraceback (most recent call last):
  File "./bug.py", line 14, in <module>
    testfun()
  File "./bug.py", line 9, in testfun
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe

./bug.py | tee -i log
^CCtrl-C pressed

answered Aug 31 '11 at 00:52

GED

161
1
4

This fixed the problem for me, but I did have to add some sys.stdout.flush() calls, which were not necessary when going to stdout only. – Chris Morlier Aug 26 '13 at 01:17
"It terminates the tee process only" -- this doesn't seem correct. When I did the experiment `./script | tee out`, the script told me it dies due to keyboard interrupt instead of broken pipe. According to [this thread](https://stackoverflow.com/a/12753167/5426033), OS does send SIGINT to all process in the group. Besides that, the proposed `-i` argument does solve what I want. Thanks! – Shichu Zhu Jun 21 '20 at 21:32

score 5 · Answer 2 · answered Jul 28 '09 at 09:25

5

This is not a python problem but a shell problem, as pointed by Brian, hitting Ctrl-C will terminate both process. Workaround is to use named pipes :

mknod mypipe p
cat mypipe | tee somefile.log &
./bug.py > mypipe

answered Jul 28 '09 at 09:25

shodanex

14,975
11
57
91

score 4 · Accepted Answer · answered Jul 28 '09 at 09:10

4

When you hit Ctrl-C, the shell will terminate both processes (python and tee) and tear down the pipe connecting them.

So when you handle the Ctrl-C in your python process and flush, it'll find that tee has been terminated and the pipe is no more. Hence the error message.

(As an aside, would you expect anything in the log ? I don't see your process outputting anything other than the flush on exit)

answered Jul 28 '09 at 09:10

Brian Agnew

268,207
37
334
440

1

I expect Ctrl-C pressed to appear in the log. – shodanex Jul 28 '09 at 09:16
Of course. Unfortunately the 'tee' process will probably have gone by then. I would simply redirect the python std out to a file, and then tail that file. Or, depending on your shell, you can probably do something fancy to redirect and still write to the console. – Brian Agnew Jul 28 '09 at 09:35
Ah. Just seen your named pipes business below – Brian Agnew Jul 28 '09 at 09:36

python stdout flush and tee

3 Answers3