Python safeiy capture live output from multiple subprocesses

Question

It is explained in https://stackoverflow.com/a/18422264/7238575 how one can run a subprocess and read out the results live. However, it looks like it creates a file with a name test.log to do so. This makes me worry that if multiple scripts are using this trick in the same directory the test.log file might well be corrupted. Is there a way that does not require a file to be created outside Python? Or can we make sure that each process uses a unique log file? Or am I completely misunderstanding the situation and is there no risk of simultaneous writes by different programs to the same test.log file?

Answer 1

You don't need to write the live output to a file. You can write it to simply to STDOUT with sys.stdout.write("your message").

On the other hand you can generate unique log files for each process:

import os
import psutil

pid = psutil.Process(os.getpid())
process_name = pid.name()
path, extension = os.path.splitext(os.path.join(os.getcwd(), "my_basic_log_file.log"))
created_log_file_name = "{0}_{1}{2}".format(path, process_name, extension)
print(created_log_file_name)

Output:

>>> python3 test_1.py
/home/my_user/test_folder/my_basic_log_file_python3.log

If you see the above example my process name was python3 so this process name was inserted to the "basic" log file name. With this solution you can create unique log files for your processes.

You can set your process name with the setproctitle.setproctitle("my_process_name"). Here is an example.

import os
import psutil
import setproctitle

setproctitle.setproctitle("milan_balazs")

pid = psutil.Process(os.getpid())
process_name = pid.name()
path, extension = os.path.splitext(os.path.join(os.getcwd(), "my_basic_log_file.log"))
created_log_file_name = "{0}_{1}{2}".format(path, process_name, extension)
print(created_log_file_name)

Output:

>>> python3 test_1.py
/home/my_user/test_folder/my_basic_log_file_milan_balazs.log

Previously I have written a quite complex and safe command caller which can make live output (not to file). You can check it:

import sys
import os
import subprocess
import select
import errno


def poll_command(process, realtime):
    """
     Watch for error or output from the process
    :param process: the process, running the command
    :param realtime: flag if realtime logging is needed
    :return: Return STDOUT and return code of the command processed
    """

    coutput = ""
    poller = select.poll()
    poller.register(process.stdout, select.POLLIN)

    fdhup = {process.stdout.fileno(): 0}
    while sum(fdhup.values()) < len(fdhup):
        try:
            r = poller.poll(1)
        except select.error as err:
            if not err.args[0] == errno.EINTR:
                raise
            r = []
        for fd, flags in r:
            if flags & (select.POLLIN | select.POLLPRI):
                c = version_conversion(fd, realtime)
                coutput += c
            else:
                fdhup[fd] = 1

    return coutput.strip(), process.poll()


def version_conversion(fd, realtime):
    """
    There are some differences between Python2/3 so this conversion is needed.
    """
    c = os.read(fd, 4096)
    if sys.version_info >= (3, 0):
        c = c.decode("ISO-8859-1")
    if realtime:
        sys.stdout.write(c)
        sys.stdout.flush()
    return c


def exec_shell(command, real_time_out=False):
    """
    Call commands.
    :param command: Command line.
    :param real_time_out: If this variable is True, the output of command is logging in real-time
    :return: Return STDOUT and return code of the command processed.
    """

    if not command:
        print("Command is not available.")
        return None, None

    print("Executing '{}'".format(command))
    rtoutput = real_time_out

    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, return_code = poll_command(p, rtoutput)

    if p.poll():
        error_msg = "Return code: {ret_code} Error message: {err_msg}".format(
            ret_code=return_code, err_msg=out
        )

        print(error_msg)

    print("[OK] - The command calling was successful. CMD: '{}'".format(command))

    return out, return_code


exec_shell("echo test running", real_time_out=True)

Output:

>>> python3 test.py
Executing 'echo test running'
test running
[OK] - The command calling was successful. CMD: 'echo test running'

I hope my answer answers your question! :)