Logging separate files for different Processes in Python

Question

I've seen a few questions regarding putting logs from different processes together when using the multiprocessing module in Python. I would like to do the opposite, produce separate log files for different processes, and they should log everything that happens when calling other modules without being mangled. In the example below I have a main program (main.py) and two modules (module1.py and module2.py), and I want the main logger (mainlog) to write to stdout, which it does fine. I also want a separate file for each process including logging from module1 and module2.

main.py:

import logging
import multiprocessing as mpr
import module1
import sys

mainlog = logging.getLogger("main")
h = logging.StreamHandler(sys.stdout)
mainlog.addHandler(h)
logging.root.setLevel(logging.DEBUG)

for i in xrange(0,3):
    mainlog.info("Starting process ... %s", i)
    log = logging.getLogger("module1")
    h = logging.FileHandler("process_{0}.log".format(i))
    fmt = logging.Formatter(fmt="%(levelname)-10s:%(filename)-20s:%(message)s")
    h.setFormatter(fmt)
    log.addHandler(h)
    log.setLevel(logging.DEBUG)
    p = mpr.Process(target=module1.do_something, args=(i,))
    p.start()

A module1.py:

import logging
import module2

log = logging.getLogger("module1")

def do_something(i):

    for j in xrange(0,100):
        log.debug("do something. process %2s. iteration %2s", i,j)
        module2.multiply(j,2)

And a module2.py:

import logging
log = logging.getLogger("module2")

def multiply(x,y):
    log.debug("... multiplying %s x %s = %s", x,y, x*y)
    return x*y

Instead I get the following output:

Starting process ... 0
Starting process ... 1
No handlers could be found for logger "module2"
Starting process ... 2
No handlers could be found for logger "module2"
No handlers could be found for logger "module2"

And 3 individual logging files (process_0.log, ...) that contain the messages from all processes together, instead of only one. Nothing from module2.py is logged. What am I doing wrong?

score 1 · Answer 1 · answered Aug 14 '13 at 23:22

You need to configure logging in the child processes. They start off with a clean slate, and logging isn't configured in them. Are you on Windows, by any chance? Almost nothing is inherited from the parent process to the child process in Windows, whereas on POSIX, fork() semantics can allow some things to be inherited.

score 1 · Accepted Answer · answered Aug 15 '13 at 17:48

I ended up creating a subclass of logging.Logger to manage switch between logging to main and logging to disk. Now I can switch when necessary inside a Process:

import logging
import sys


class CGLogger(logging.Logger):
    def __init__(self,name):
        logging.Logger.__init__(self,name)
        self.mainhandler = logging.StreamHandler(sys.stdout)
        self.addHandler(self.mainhandler)

    def stop_main_logging(self):
        self.removeHandler(self.mainhandler)

    def log_to_file(self, fn):
        self.filehandler = logging.FileHandler(fn)
        self.addHandler(self.filehandler)

    def stop_logging_to_file(self):
        self.removeHandler(self.filehandler)

    def restart_main_logging(self):
        self.addHandler(self.mainhandler)

    def switch_to_file_logging(self, fn):
        self.stop_main_logging()
        self.log_to_file(fn)

    def switch_to_main_logging(self):
        self.stop_logging_to_file()
        self.restart_main_logging(fn)

logging.setLoggerClass(CGLogger)

Can you share a working example? I didn't manage to make this work — Carlos Azevedo, May 12 '20 at 22:24

Logging separate files for different Processes in Python

2 Answers2

Linked