subprocess run in background and write output line by line to a file

Question

I have two files:

main.py

import subprocess
import shlex


def main():
    command = 'python test_output.py'
    logfile =  open('output', 'w')
    proc = subprocess.Popen(shlex.split(command), stdout=logfile)


if __name__ == "__main__":
    main()

and test_output.py

from time import sleep
import os

for i in range(0, 30):
    print("Slept for => ", i+1, "s")
    sleep(1)
os.system("notify-send completed -t 1500")

The output of the process is written in logfile once the child process is completed. Is there any way to:

1. Start child process from main and exit it (like it does now).
2. Keep running the child process in background.
3. As child process produces an output, write it immediately to logfile. (Don't wait for the child process to finish, as it does now.)

There are other questions (like this one) where solution is given for reading line by line, but they make the main.py wait. Is it possible to do everything in background, without keeping main.py waiting?

Answer 1

Both the buffers of the filehandler as the subprocess can be set to 'line-buffering', where a newline character causes each object's buffer to be forwarded. This is done by setting the buffer parameter to 1, see open() command and subprocess.

You need to make sure that the child process will not buffer by itself. By seeing that you are running a Python script too, you either need to implement this in the code there, like flush=True for a print statement:

print(this_and_that, flush=True)

Source

Credit