1

I am writing a little Python script to log the output and error of another Python program, and simultaneously print them on my screen as usual. This code works well, except it buffers output of the subprocess python program and output them together once for a long time, instead of printing instantly as usual:

subprocess.run(f"python {args.launch} 2>&1 | tee -a {file_path}", shell=True)

in which args.lunch is the name of my python program and file_path is the log file. I have tried some existing solutions but none works for me.

tripleee
  • 175,061
  • 34
  • 275
  • 318
Jojo Zhai
  • 51
  • 3
  • 2
    try running the script with `python -u $yourscript` or set enviroment variable PYTHONUNBUFFERED and see if it helps.. – rasjani Mar 17 '23 at 10:30

1 Answers1

0

As mentioned by @rasjani, one way around this is to use the -u flag

subprocess.run(f"python -u {args.launch} 2>&1 | tee -a {file_path}", shell=True)

I don't have a good explanation for why this works.

edit - edited to remove incorrect second option

FiddleStix
  • 3,016
  • 20
  • 21
  • 1
    Google "output buffering". The OS buffers output when it goes into a pipe for performance reasons, so instead of writing every few bytes as soon as they arrive, it waits for an I/O buffer to fill up before writing anything. This is an extremely common FAQ and the rationale for the `-u` option. – tripleee Mar 17 '23 at 11:28
  • 1
    The "alternatively" part is completely wrong. You are passing `'2>&1'`, `'|'`, `'tee'` etc as arguments to `python`. If you want to use shell features like redirection or pipes, you need a shell. See also [Actual meaning of `shell=True` in subprocess](https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess) – tripleee Mar 17 '23 at 11:29
  • @tripleee Thanks a lot for your explanation, which solves all my doubts. – Jojo Zhai Mar 17 '23 at 14:51
  • @FiddleStix Thanks a lot and the first solution works. As what tripleee says, the second dose not save anything, though it can output instantly. Does anybody know why the second can make output instant? – Jojo Zhai Mar 17 '23 at 15:01
  • 1
    Presumably `args.launch` ignores the spurious arguments. But that means `tee` doesn't get run at all, nor is any redirection taking place, so neither is there any pipe, and so no output buffering. – tripleee Mar 17 '23 at 20:35
  • @tripleee I'm aware of what output buffering is but I'd be curious on your thoughts about a) how Python knows whether it is being called interactively or not and b) why a bash program (i.e. a bash script with an echo, a sleep and another echo) does not buffer when Python does. – FiddleStix Mar 20 '23 at 16:38
  • Generally, it's the OS which decides whether to buffer or not. What exactly each shell does in this situation is probably too broad a topic to fit into this comment, but if you see them not buffer when attached to a pipe, chances are they explicitly select unbuffered output. – tripleee Mar 20 '23 at 16:47