148

I made a python script which communicates with a web server using an infinite loop. I want to log every communication data to a file and also monitor them from terminal at same time. so I used tee command like this.

python client.py | tee logfile

however, I got nothing from terminal nor logfile. the python script is working fine. what is happening here? am I missing something?

some advice would be appreciated. thank you in advance.

daehee
  • 5,047
  • 7
  • 44
  • 70
  • 4
    Buffering behaves differently for pipes and terminals. You may need to do an explicit `sys.stdout.flush()` from your script whenever you logged a line. – Lukas Graf Feb 09 '14 at 18:04
  • For other ways to trigger unbuffered output see http://stackoverflow.com/q/107705/1328439 – Dima Chubarov Mar 31 '17 at 09:11

2 Answers2

252

From man python:

   -u     Force stdin, stdout and stderr to  be  totally  unbuffered.   On  systems
          where it matters, also put stdin, stdout and stderr in binary mode.  Note
          that there is internal buffering in xreadlines(), readlines()  and  file-
          object  iterators  ("for  line  in sys.stdin") which is not influenced by
          this option.  To work around this, you will want to use  "sys.stdin.read‐
          line()" inside a "while 1:" loop.

So what you can do is:

/usr/bin/python -u client.py >> logfile 2>&1

Or using tee:

python -u client.py | tee logfile
Vor
  • 33,215
  • 43
  • 135
  • 193
  • 1
    An alternative would be to use `script`, which also disables buffering and additionally makes control sequences (`C-a`, cursor keys etc) work: http://stackoverflow.com/a/39269661/15690. – blueyed Sep 05 '16 at 13:45
  • excellent! it worked also in Python 3 on my Raspberry Pi 3 equipped with Raspbian Jessie: python3 -u client.py | tee logfile – Antonino Mar 27 '17 at 04:22
  • A note: python, like various other commands will use line buffered if stdin and stdout are consoles, but full buffered if the results are redirected to a file or a pipe. `tee` is seen like a pipe (which it is) and not the hybrid: it write to console. Note: the behaviour could be also controlled inside a python program. – Giacomo Catenazzi Apr 06 '19 at 18:28
  • Another note: `python -u client.py | tee >> logfile` would not work. The `>>` would introduce another case of buffered writing to a file. That's what `tee -a` solves. – tanius Jun 26 '19 at 20:16
  • One can also put -u into the shebang line and it works: #!/usr/bin/python3 -u – Sergey M Apr 28 '21 at 03:52
3

Instead of making it fully unbuffered you can make it linebuffered as it is normally with sys.stdout.reconfigure(line_buffering=True) (after import sys of course).

This was added in 3.7, docs: https://docs.python.org/3/library/io.html#io.TextIOWrapper.reconfigure

stefanct
  • 2,503
  • 1
  • 28
  • 32