Python subprocess delayed in screen buffer while writing print logs in file

Question

Below is my script which calls another script using below example and produces entire output after execution is completed while expectation is that it should produce live output as well as write in file at the same time.
What happening is that its not producing output line by line and printing entire output in the end.

./My_Tests.py ALL
TC 1 : PASSED
TC 2 : PASSED
Tatal Passed Tests : 2
Tatal Failed Tests : 0

My_Tests.py :

from subprocess import Popen, PIPE, STDOUT
with Popen("./tests.py", stdout=PIPE, stderr=STDOUT, bufsize=1, universal_newlines=True) as p, \  
    open('tests.log', 'ab') as file:  
    for line in p.stdout: # b'\n'-separated lines  
        print(line, end='') #new addition  
        sys.stdout.buffer.write(line)  
        file.write(line)

I also tried with below command but it prints only on Terminal and not saving output in file.
import subprocess

# Run command and redirect it by | tee to a file named out.txt 
p = subprocess.Popen([command, '|', 'tee', 'out.txt'])
p.wait()

did u get chance to look on https://stackoverflow.com/questions/15535240/python-popen-write-to-stdout-and-log-file-simultaneously — pankaj mishra, Dec 27 '17 at 05:59
Thanks yes i had looked at it but i was stuck at how i can use it for my issue. Thanks for the tip. — Ujjawal Khare, Dec 30 '17 at 19:31

score 0 · Accepted Answer · answered Dec 28 '17 at 08:10

What was the issue and solution :
My tests.py file had many print statements what i was doing wrong:

I was calling tests.py file directly inside above code which was causing whole file to execute obviously and then print the output.

Here is what i did :
1. Created a function for above subprocess code.
2. Created functions for every part of code inside my tests.py and made n numbers of tests-n.py
3. Then call thiese tests-n.py inside subprocess function code one after another. Parent file is now tests.py

    from subprocess import Popen, PIPE, STDOUT  

    def logc(file, logfile):
    with Popen(file, stdout=PIPE, stderr=STDOUT, bufsize=1, universal_newlines=True) as p, \  
        open(logfile, 'ab') as file:  
        for line in p.stdout: # b'\n'-separated lines  
            print(line, end='') #new addition  
            sys.stdout.buffer.write(line)  
            file.write(line)  

    os.system("> logs.py")
    logc("tests1.py", "logs.py")
    logc("tests2.py", "logs.py")
    logc("tests3.py", "logs.py")
    logc("tests3.py", "logs.py")

Python subprocess delayed in screen buffer while writing print logs in file

1 Answers1

Linked