python output system calls to terminal and logfile

Question

I would like to execute system calls from my (python) script such that the output of the sys-call is generated on the terminal where script is running as well as is captured in a logfile. However I am not able to make this work for interactive system calls.

I started with following code that does not capture the system call in a logfile, but works correctly (as in is able to display output on terminal, and can take inputs I type in terminal) for both basic commands such as system('echo HEYA') as well as interactive commands such as system('python') :

def system(cmd):
    log.info(f"Running: {cmd}")
    try:
        subprocess.run(cmd,  check=True, shell=True, executable=SHELL)
    except subprocess.CalledProcessError as err:
        log.error(err.output)

Notes: log is a logger (created using standard logging module). SHELL variable holds a shell of my choice.

Now, I modify the above to be able to redirect the process output to terminal as well as logfile on disk in realtime with following code:

def system(cmd):
    log.info(f"Running: {cmd}")
    try:
        process = subprocess.Popen(cmd, 
                             shell=True, executable=SHELL, universal_newlines=True, 
                             stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        while True:
            o = process.stdout.readline().rstrip('\n')
            if o == ''  and process.poll() is not None:
                break
            if o:
                syslog.info(o)
        ret = process.poll()
        if ret:
            log.error(f"Failed with exit code: {ret}")
        else:
            log.info("Done")
    except:
        err = sys.exc_info()[0]
        log.error(err)
        raise

Notice that I use a different logger (syslog) to redirect outputs to logfile and terminal. The only reason for this, is I want system command outputs formatted differently from other messages generated in the script.

The second version of system function works for something like system('echo HEYA') but not for interactive calls like system('python'). Any suggestions what I may be doing wrong, and how I can get this to work?

score 0 · Answer 1 · answered Sep 12 '21 at 06:02

Based on an earlier post of similar nature: https://stackoverflow.com/a/651718/5150258

I was able to get this work partially by using first form of system definition with a tee:

tee = subprocess.Popen(["tee", "sys_cmds.log"], stdin=subprocess.PIPE)
os.dup2(tee.stdin.fileno(), sys.stdout.fileno())
os.dup2(tee.stdin.fileno(), sys.stderr.fileno()
def system(cmd):
    log.info(f"Running: {cmd}")
    try:
        subprocess.run(cmd,  check=True, shell=True, executable=SHELL)
    except subprocess.CalledProcessError as err:
        log.error(err.output)

However this isn't the perfect solution since I lose formatting control of the output as it goes directly without passing through the logger object.

python output system calls to terminal and logfile

1 Answers1