Logging printout of an executed python file within another file and printing out the result in terminal simultaneously

Question

I have two Python files (main.py and main_test.py). The file main_test.py is executed within main.py. When I do not use a log file this is what gets printed out:

Main file: 17:41:18
Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20

When I use a log file and execute main.py>log, then I get the following:

Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20
Main file: 17:41:18

Also, when I use python3 main.py | tee log to print out and log the output, it waits and prints out after finishing everything. In addition, the problem of reversing remains.

---

Questions

1. How can I fix the reversed print out?
2. How can I print out results simultaneously in terminal and log them in a correct order?

---

Python files for replication

main.py

import os 
import time
import datetime
import pytz

python_file_name = 'main_test'+'.py'

time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone

curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()

print(f'Main file: {curr_time}')
cwd = os.path.join(os.getcwd(), python_file_name)
os.system(f'python3 {cwd}')

main_test.py

import pytz
import datetime
import time

time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone

for i in range(3):
    curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()
    print(f'Executed file: {curr_time}')
    time.sleep(1)

Answer 1

When you run a script like this:

python main.py>log

The shell redirects output from the script to a file called log. However, if the script launches other scripts in their own subshell (which is what os.system() does), the output of that does not get captured.

What is surprising about your example is that you'd see anything at all when redirecting, since the output should have been redirected and no longer echo - so perhaps there's something you're leaving out here.

Also, tee waits for EOF on standard in, or for some error to occur, so the behaviour you're seeing there makes sense. This is intended behaviour.

Why bother with shells at all though? Why not write a few functions to call, and import the other Python module to call its functions? Or, if you need things to run in parallel (which they didn't in your example), look at multiprocessing.

In direct response to your questions:

• "How can I fix the reversed print out?"
  Don't use redirection, and write to file directly from the script, or ensure you use the same redirection when calling other scripts from the first (that will get messy), or capture the output from the subprocesses in the subshell and pipe it to the standard out of your main script.

• "How can I print out results simultaneously in terminal and log them in a correct order?"
  You should probably just do it in the script, otherwise this is not a really a Python question and you should try SuperUser or similar sites to see if there's some way to have tee or similar tools write through live.

In general though, unless you have really strong reasons to have the other functionality running in other shells, you should look at solving your problems in the Python script. And if you can't, use you can use something like Popen or derivatives to capture the subscript's output and do what you need instead of relying on tools that may or may not be available on the host OS running your script.