5

Is there a way to increase the stdout buffer size from 8182 in Python or to delay the flush until I actually call flush?

Things I've tried that don't work:

  • I can get around this issue on Windows because I can access the buffer directly (e.g. see my answer to this post). But this doesn't work for Unix.
  • I can increase the buffer size for a file by passing buffer to the constructor, however stdout is already constructed.
  • Turning off buffering (python -u) obviously makes things worse!
  • Using a temporary buffer encounters the same problems - stdout is flushed after every 8192nd byte is copied from the temporary buffer.

Rationale: The aim here is to reduce console flickering. Buffering everything, as per this question indeed works, for instance when I try in C or by using the Windows API in Python, but the 8182 limit in Python seems to be causing problems that I can't get around on Unix.

c z
  • 7,726
  • 3
  • 46
  • 59
  • 1
    I'd like to know why you need to do this :) – AKX Nov 26 '20 at 11:27
  • 2
    Why do you want to do this? It sounds like you might be better served by some other approach, such as doing your own buffering. – user2357112 Nov 26 '20 at 11:27
  • 1
    "Using a temporary buffer encounters the same problems when the temporary is ultimately copied to stdout." - what problems? – user2357112 Nov 26 '20 at 11:29
  • 2
    It sounds like you should have asked a question about avoiding terminal flickering, not a question about buffer size adjustment. – user2357112 Nov 26 '20 at 11:36
  • 2
    https://stackoverflow.com/a/107717/7256228 – zaidfazil Nov 26 '20 at 11:37
  • @user2357112supportsMonica Re: *"should have asked a question about avoiding terminal flickering"* - [That question](https://stackoverflow.com/questions/34842526/update-console-without-flickering-c) is what led me here haha. I've updated my question with the source, but maybe this would be useful for someone else anyway, I presume the `FileIO` `buffer` argument exists for a reason. – c z Nov 26 '20 at 12:18
  • @cz I was referring the other thing explained in that answer. To quote `You could also replace sys.stdout with some other stream like wrapper which does a flush after every call.` – zaidfazil Nov 27 '20 at 14:42

2 Answers2

5

You can wrap the raw stdout stream, available as sys.stdout.buffer, with a larger buffer size with io.BufferedWriter, and then wrap the resulting buffered binary stream as a buffered text stream with io.TextIOWrapper:

import io
import sys

sys.stdout = io.TextIOWrapper(io.BufferedWriter(sys.stdout.buffer, new_size))

Here's a demonstration of the effect of increasing the buffer size of stdout to 100000 so that it does not flush the two 10000-character-long print outputs until a manual flush is called:

import io
import sys
import time

print('Original buffer size:', io.DEFAULT_BUFFER_SIZE)
for large_buffer in False, True:
    if large_buffer:
        print('Increasing buffer size...')
        sys.stdout = io.TextIOWrapper(io.BufferedWriter(sys.stdout.buffer, 100000))
    for i in range(2):
        time.sleep(2)
        print(str(i * 2) * 10000)
        time.sleep(2)
        print(str(i * 2 + 1) *10000)
        print(f'Flush #{i + 1}')
        sys.stdout.flush()

Demo: https://repl.it/@blhsing/UnkemptGullibleDecompiler

blhsing
  • 91,368
  • 6
  • 71
  • 106
  • Thanks, but this just holds the data in `BufferedWriter` until you call `sys.stdout.flush`. When `BufferedWriter` pushes to the *real* stdout, it still has same underlying problem - it flushes for every 8182nd byte written (from the `BufferedWriter`). That said I notice this considerably improves application performance (if not the flickering) when writing to a slow terminal such as in PyCharm. – c z Nov 27 '20 at 09:12
4

Found the answer, actually very simple:

my_stdout = open( 1, "w", buffering = 100000 )
  • 1 is the fileno for stdout.
  • sys.stdout = my_stdout can be used to make the change to the default print target.
  • I've only tested this on Unix.
c z
  • 7,726
  • 3
  • 46
  • 59