Python and Bash - how to "print" output in the file and "tail" it while it is being executed

Question

I have my python script with many print statements in it. It does some job that takes more than a day, so I need to track log in the output file periodically while it is being executed. if I execute

python migrate.py > output.txt

my file gets populated only after the script is finished (after 24 hours) which doesn't suite me. I tried as well with:

python migrate.py  2>&1 | tee output.txt

But in this case:

• nothing gets printed in the terminal while this is being executed
• tail command provides no output until script finishes...

I want to be able to see with tail content of my output file but the size of the file is always 0 until the script is finished. Thanks a lot!

Edit: example of my code (if this is relevant, but I do not think so)

............
.............
.............
def main():
    """
    Script main function
    """
    start_time = datetime.datetime.now()
    print ("----------------------------------------------------------------------------")
    print ("### script started ###")
    args = get_args()
    print ("depot:"+depot)
    print ("stream:"+args[0])
    print ("destination:"+args[1])

    print ("----------------------------------------------------------------------------")
    
    logfile = get_history(args[0])
    print ("----------------------------------------------------------------------------")
    
    git_migrate(logfile, args[0], args[1], args[2], args[3], depot)
    print ("----------------------------------------------------------------------------")        

        
    print ("### script completed ###")    
    end_time = datetime.datetime.now()
    print('Duration: {}'.format(end_time - start_time))


if __name__ == '__main__':
    main()

Answer 1

The output sent to stdout (whether by print() or by something like sys.stdout.write()) is accumulating in a buffer. You can get it to periodically write by flushing that buffer with sys.stdout.flush():

import time
import sys

for x in range(1000):
    time.sleep(5)

    # Either sys.stdout.write or print works here
    print(x)  # sys.stdout.write(str(x))
    if x % 5 == 0:
        sys.stdout.flush()

This writes to the file where output is being redirected every five loop iterations.

In your code, you just would put sys.stdout.flush() at whichever points you would like to be able to see progress. (That may or may not be inside a loop; the one in my example is just to generate code that runs for a bit.)

Answer 2

As @matszwecja said file.write is the best option for you. Try it yourself:

count = 0
file = open("file.txt", "a")

while(True):
    file.write(str(count) + "\n")
    count += 1
    if count == 34:
        count = 0

with this simple code you can see that while loop is running you can execute tail in another tab and it will give you output, so just change prints for file.writes (or puth both) and it will do the job.

in your code:

def main():
    """
    Script main function
    """
    file = open("log.log", "a")
    start_time = datetime.datetime.now()
    file.write ("----------------------------------------------------------------------------")
    file.write ("### script started ###")
    args = get_args()
    file.write ("depot:"+depot)
    file.write ("stream:"+args[0])
    file.write ("destination:"+args[1])

    file.write ("----------------------------------------------------------------------------")
    
    logfile = get_history(args[0])
    file.write ("----------------------------------------------------------------------------")
    
    git_migrate(logfile, args[0], args[1], args[2], args[3], depot)
    file.write ("----------------------------------------------------------------------------")        

        
    file.write ("### script completed ###")    
    end_time = datetime.datetime.now()
    file.write('Duration: {}'.format(end_time - start_time))

This way you will be able to use tail to see the file while it's being written.