1

I am running another python background process within a python script and am trying to pipe its stdout to a file. I am running:

with open("log.log", "wb") as out:
    p = subprocess.Popen(["python", "main.py"], stdout=out, bufsize=1)

However, the file is always empty while I can see in shell the output of the popen process just fine. Any idea?

fsociety
  • 1,791
  • 4
  • 22
  • 32
  • Do you want to save main.py output in the log.log file? – omides248 May 04 '21 at 13:09
  • Yes indeed @omides248 – fsociety May 04 '21 at 13:14
  • This is standard buffering. `wait` for the process and it will close the file, which flushes output through to the disk. But probably don't use `Popen` at all if you can avoid it; as suggested in the `subprocess` documentation, prefer `run` when you can. Then you don't need to `wait` explicitly because `run` takes care of managing the `Popen` object properly and shutting it down. – tripleee May 12 '21 at 12:45

1 Answers1

1

You can use buffering=1 for write realtime to file. python write to file in real time

and use w+ for append

test.py:

import subprocess

if __name__ == '__main__':
    
    with open("log.log", "w+", buffering=1) as f:
        p = subprocess.Popen(["python", "main.py"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
        for line in p.stdout:
            f.write(line.decode("utf-8"))

main.py:

import random
import time

if __name__ == '__main__':

    while True:
        print(f"Hello {random.randint(1, 100)}")
        time.sleep(1)
omides248
  • 128
  • 2
  • 12
  • `shell=True` is an error when the first argument is a list. Weirdly, it works on Windows; but it's still a bug and a very common error. – tripleee May 12 '21 at 12:49
  • And having Python read and then write the output is just inefficient; the OP's original code was better. – tripleee May 12 '21 at 12:50
  • Assuming UTF-8 encoding hampers portability; there is no particular reason here to not simply read and write `bytes`, which will be more efficient, too. (But the proper fix is to do neither; see the previous comment.) – tripleee May 12 '21 at 12:52