7

Using following code:

import sys

print("INFO", flush=True, file=sys.stdout)
print("ERROR", flush=True, file=sys.stderr)

Sometimes the output is:

ERROR
INFO

and sometimes

INFO
ERROR

How can one make it to always print by the order written in the code?

Looping Ydobon
  • 71
  • 3

2 Answers2

1

using flush=True only makes python flush the data in its "application buffer" to the "OS buffers", but it doesn't ensure that the operating system buffers are flushed to disk/screen.

if you are outputting the results to a file (or using a few IDEs), then in order to flush those buffers you have to issue a system call using os.fsync which calls the OS to flush its own buffers to their final destination (disk/screen), this is a slow operation which involves a system call so it shouldn't be used carelessly after every print.

import sys
import os

print("INFO", flush=True, file=sys.stdout)
os.fsync(sys.stdout.fileno())
print("ERROR", flush=True, file=sys.stderr)
os.fsync(sys.stderr.fileno())

in case you were running this in a terminal you should wrap os.fsync in a try/except block for OSError (preferably with contextlib.suppress(OSError)).

note that some IDEs implement their own logic for reading file descriptors, so it may not work for all IDEs.

For pycharm on linux/macos there is a bug that was reported in 2013 and still not fixed about this problem (because it's practically not fixable), a workaround is to direct stderr to stdout, and lose the red coloring, this is applicable to any IDE that manually handles stdout buffers and hence produces this bug. (or sleep for 1 millisecond to force a context switch by the OS, but that's not practical)

import sys
import os
os.dup2(sys.stdout.fileno(),sys.stderr.fileno())  # redirect os stderr buffers to stdout
sys.stderr = sys.stdout  # redirect python stderr buffers to stdout

print("INFO", flush=True, file=sys.stdout)
print("ERROR", flush=True, file=sys.stderr)
Ahmed AEK
  • 8,584
  • 2
  • 7
  • 23
  • 1
    This appears to be correct; awaiting confirmation from others before awarding the bounty. – Karl Knechtel Jan 06 '23 at 22:14
  • @KarlKnechtel how is this correct because the error statement still prints first some of the time? – Life is complex Jan 07 '23 at 06:36
  • @Lifeiscomplex i tried windows and linux on all of pycharm/jupyter/terminal/spyder and they are all printing in the correct order for 50+ times ... what IDE are you using ? – Ahmed AEK Jan 07 '23 at 07:29
  • @AhmedAEK I'm using a Mac with pycharm and Python 3.9.x. I tested your updated code and it still does not print in order consistently. – Life is complex Jan 07 '23 at 13:41
  • @Lifeiscomplex yeh that's a bug in pycharm, so you can only redirect stdout to stderr, and lose the coloring. – Ahmed AEK Jan 07 '23 at 14:36
  • @AhmedAEK losing the coloring isn't the problem. Both pieces of code do not print in a consistent order in my environment. – Life is complex Jan 07 '23 at 14:38
  • @Lifeiscomplex pretty sure the current last snippet with `dup2` will work correctly even if your were printing to stderr from an external C library. – Ahmed AEK Jan 07 '23 at 15:11
-1

The order in which you write the statements in your code does not guarantee the order in which they will be printed. This is because stdout and stderr are two different streams and the order in which they're written to can be affected by various factors such as buffering and output device performance.

If you want to ensure that the output is always printed in a specific order, you can use a lock or mutex to synchronise access to the streams. This will ensure that only one thread can write to the streams at a time, so the output will be printed in the order that the threads acquire the lock. Using your code as an example, give this a try -

import threading
import sys

lock = threading.Lock()

def print_info():
    with lock:
        print("INFO", flush=True, file=sys.stdout)

def print_error():
    with lock:
        print("ERROR", flush=True, file=sys.stderr)

print_info()
print_error()
Doomian
  • 9
  • 3
  • As an extension: is it possible to control the order if the program is only explicitly writing to stdout, whereas the stderr output comes from exception tracebacks? (The bounty was prompted by a question involving such a situation.) – Karl Knechtel Jan 02 '23 at 15:22
  • @KarlKnechtel, isn't the print order maintained when the error comes from exception tracebacks? – anirudh sharma Jan 04 '23 at 05:13
  • @anirudhsharma please see the "Linked" questions in the sidebar (the ones I hoped to send here as a duplicate) for examples of issues others have had with that. – Karl Knechtel Jan 04 '23 at 05:14
  • 1
    I tested your answer in my environment, which is python 3.9.x and it does not work correctly. The `sys.stderr` will occasionally print before `sys.stdout` does. – Life is complex Jan 04 '23 at 14:00
  • I don't think that locks work like this – TheTechRobo the Nerd Jul 13 '23 at 22:36