redirect sys.stdout to a file without buffering in Python 3

Question

I have a script that writes to a log file. In Python 2, my quick solution to allow tailing/viewing of the log as it progressed was by assigning sys.stdout to a file object with buffering set to 0:

original_stdout = sys.stdout
sys.stdout = open(log_file, 'w', 0)

Once set, any print statements in the script's functions redirect to the log file very nicely.

Running the 2to3-converted version under Python 3 gives the following error: ValueError: can't have unbuffered text I/O. Changing the 'w' above to 'wb' solves that, so the structure of the block is

original_stdout = sys.stdout
sys.stdout = open(log_file, 'wb', 0)
print("{}".format(first_message))

but now the first print statement errors with TypeError: 'str' does not support the buffer interface. I tried explicitly casting the string to bytes

print(bytes("{}".format(first_message), "UTF-8"))

but that produces the same TypeError as before.

What is the easiest way to write unbuffered text to a file in Python 3?

Answer 1

According to Python 3.4.3 documentation at https://docs.python.org/3/library/io.html#raw-i-o and 3.5 documenmtation at https://docs.python.org/3.5/library/io.html#raw-i-o the way to get unbuffered IO is with Raw IO which can be enabled as in:

f = open("myfile.jpg", "rb", buffering=0)

That means "wb" should work for writing.

Details on Raw IO are at https://docs.python.org/3/library/io.html#io.RawIOBase and https://docs.python.org/3.5/library/io.html#io.RawIOBase which appear to be the same.

I did some testing and found buffering of Text IO to be severe and can amount to hundreds of lines and this happens even when writing to sys.stderr and redirecting the error output to a file, on Windows 7 at least. The I tried Raw IO and it worked great! - each line printed came through immediately and in plain text in tail -f output. This is what worked for me on Windows 7 with Python 3.4.3 and using tail bundled with GitHub tools:

import time
import sys
f = open("myfile.txt", "ab", buffering=0)
c = 0
while True:
    f.write(bytes("count is " + str(c) + '\n','utf-8'))
    c += 1
    time.sleep(1)

Answer 2

If by unbuffered you mean having the outputs immediately flushed to disk, you can simply do this:

original_stdout = sys.stdout
sys.stdout = open(log_file, 'w')
print(log_message, flush=True)

As print is now a first-class function you can also specify which file to print to, such as:

fd = open(log_file, 'w')
print(log_message, file=fd, flush=True)

Answer 3

The issue seems to be in the way you open the file -

open(log_file, 'w', 0)

From Python 3.x documentation -

open(file, mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)

The third argument for open() determines the buffering mode for the file, 0 means no buffering. I do not think you can make it work by just using 'wb' instead of 'w' .

You should remove that 0 third argument, and let open() use default line buffering for text files. Example -

open(log_file, 'w')