When using subprocess.call(), how to set it in way that redirects stdout to a log file if you are using the disctionary configuration for logging?

Question

I am trying to rewrite this batch line in Python:

mkdir %TEMP%\FW >> %LOGDETAILS% 2>&1

When using subprocess.call(), how to set it in way that redirects stdout to a log file if you are using the dictionary configuration for logging?

My Python code looks like this so far (this is just part of it):

    DICT_CONFIG = {#I have my settings in here}
    logging.config.dictConfig(DICT_CONFIG)
    logdetails = logging.getLogger('SikuliScriptDetails_Logger')

    def my_function():
        logdetails.info("---------Benginning Tests--------------")
        #Set Project Name, Disable Feedback Dialogs by setting launches to 41
        returncode = subprocess.call(["regedit", "-s", "MainFolder/FwSetup.reg"], stderr = STDOUT, stdout = logdetails)

I can not test my program for quite a while until I have some other modules ready. But is my attempt at redirection correct? Is this going to be able to log the output of the ["regedit", "-s", "MainFolder/FwSetup.reg"] into my logdetails file logger?

Is it necessary for me to do stderr = STDOUT first like I did before sending stdout to logdetails file logger?

Also, does anyone know what the 41 means? I do not know what to do with it.

PS: I've looked at this already, but I still don't get it since they are not using dictionary configuration for logging.

UPDATE: PS: I've also looked at this information to understand the batch line of code.

UPDATE2: Sorry, I gave you the wrong line of code at the beginning. The batch file I was supposed to give paste here is:

"regedit", "-s", "VBoxScripts/FwSetup.reg"

not this one: mkdir %TEMP%\FW >> %LOGDETAILS% 2>&1 Thanks.

Answer 1

Your question is a bit confused.

---

First, there's the logging bit. It doesn't matter which mechanism you use for configuring logging; in the end, you still end up with a Logger object with the same methods. What you're doing seems reasonable, although it is a bit weird to use 'SikuliScriptDetails_Logger' as a name for a logger.

---

Next, there's this:

Is it necessary for me to do stderr = STDOUT first like I did before sending stdout to logdetails file logger?

Setting stderr=STDOUT means, as the docs explain:

… that the stderr data from the child process should be captured into the same file handle as for stdout.

In other words, if you want to log stdout and stderr together, then yes, you do need to do this; if you want to log just stdout, or just stderr, or to log them each separately, then no, you should not do this.

From your batch line, it sounds like you do want to mix stdout and stderr together, in which case you're doing the right thing.

---

Finally, there's what you pass for stdout. From the same documentation:

Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None.

In particular, you cannot give it a Logger object; you have to give it a file object or PIPE.

If you just want it to append stdout and stderr to the same file the logger is using, you can get the logger's file object and pass that. However, a much simpler way to do it would be to not use logging in the first place—just open a file, write the header to that, and pass it as the stdout parameter. For example:

with open(log_details_path, 'a') as logfile:
    logfile.write("---------Benginning Tests--------------\n")
    returncode = subprocess.call(["regedit", "-s", "MainFolder/FwSetup.reg"], stdout=logfile, stderr=STDOUT)

---

But, given that you're using logging in the first place, it sounds like what you really want is something like this: read each line of stderr, and log it as a separate log message. If so, you need to use a PIPE. If you need to stream it continuously to the log (e.g., it's going to take a long time, send a lot of data, possibly fail in the middle…), you'll need to explicitly read from the PIPE and then log what you get. But if not, you can just use communicate, or even check_output. For example:

child_output = subprocess.check_output(["regedit", "-s", "MainFolder/FwSetup.reg"], stderr = STDOUT)
for line in child_output.splitlines():
    logdetails.info(line)

Answer 2

Its easy to redirect output to a file, harder to do it through the logging module. Assuming that all you really want is the python equivalent of piping stdout and stderr to a file, you can do:

log_details = "SikuliScriptDetails.log"
returncode = subprocess.call(
    ["regedit", "-s", "MainFolder/FwSetup.reg"],
    stdout=open(log_details, 'a'), 
    stderr=subprocess.STDOUT)

You can use a with clause if you worry about the file closing:

log_details = "SikuliScriptDetails.log"
with open(log_details, 'a') as log_fp:
    returncode = subprocess.call(
        ["regedit", "-s", "MainFolder/FwSetup.reg"],
        stdout=log_fp, 
        stderr=subprocess.STDOUT)

If you want to log stdout as info and stderr as warn, you can use threads:

import threading
import subprocess

def log_thread(pipe, logger):
    for line in pipe:
        logger(line.strip())

def my_call(logger, cmd, **kw):
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
        **kw)
    stdout = threading.Thread(target=log_thread, args=(proc.stdout, logger.info))
    stdout.start()
    stderr = threading.Thread(target=log_thread, args=(proc.stderr, logger.warn))
    stderr.start()
    stdout.join()
    stderr.join()
    return proc.wait()