How to write specific outputs to multiple log files using logging in a for loop in python?

Question

Goal: To run an experiment 'A' 10 times, each time with a different input. Experiment A returns a floating point value as its output (let us call it result). All the 10 results of the 10 experiments should be saved in a results file (using logging). There is a lot of logging that needs to be done while running experiment A (each time). All the intermediate outputs of the (10) experiments should be written to (10) different log files.

In other words, If my experiment directory is 'exp_dir'. There should exist a results (log) file after the end of the experiment that saves the output of each of the experiment (only the output needs to be saved, nothing else). During the process of running each experiment I will create sub directories (exp1, exp2 and so on) for each of the 10 experiments within the exp_dir and log the outcomes of the experiments in the corresponding sub directories (experiment 1 outcome should be saved in exp1 sub directory). I am planning to do this in a single python file using for loop.

Could you let me know how to do the logging separately? This is what I tried (instead of 10 experiments I tried on 2 experiments)

import logging
import os

class MyLogger(object):
    def __init__(self):
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.INFO)

    def activateLogFile(self, f):
        d =  os.path.dirname(f)
        os.makedirs(d, exist_ok=True)
        handler = logging.FileHandler(f)
        self.logger.addHandler(handler)

    def doLogging(self, fn, info):
        self.activateLogFile(fn)
        self.logger.info("{i}\n".format(i=info))


def run():

    exp_dir = os.getcwd()
    myLog.doLogging("{d}/res.log".format(d=exp_dir), "This is where results should be logged")
    myLog.logger.propagate = False
    for i in range(1, 3):
        myLog.doLogging("{d}/exp{i}/info.log".format(i=i, d=exp_dir), "some logging")
    myLog.doLogging("{d}/res.log".format(d=exp_dir), "Experiments done")

myLog = MyLogger()

if __name__ == "__main__":
    run()

But everything that is logged in exp2 is also logged in exp1 and res.log consists of all the logging (exp1, exp2 also)

I appreciate your help

Just to make it clear, the question is how to do the logging of each experiment separately and also how to save the outcome of each experiment in a results file. — user1540393, Aug 17 '17 at 18:41
You've showed us what you want to do, but you haven't shown us what you tried. Stack Overflow is not a code writing service. You need to demonstrate effort on your part before posting a problem here. You've imported logging, but we don't see anywhere you've tried to do anything with it. Have you read the documentation? Tried something and it didn't work? Edit your question and post what you have tried. And don't comment on your question; edit the question itself and add your clarification. — AJNeufeld, Aug 17 '17 at 18:55

score 1 · Answer 1 · answered Aug 17 '17 at 20:53

Your code reads (approximately):

self.logger = logging.getLogger()

handler = logging.FileHandler(f1)
self.logger.addHandler(handler)
self.logger.info("Log exp 1 stuff")   # Logs to f1

handler = logging.FileHandler(f2)
self.logger.addHandler(handler)
self.logger.info("Log exp 2 stuff")   # Logs to f1, f2

handler = logging.FileHandler(f3)
self.logger.addHandler(handler)
self.logger.info("Log global stuff")   # Logs to f1, f2, f3

Note you are adding a second, and then a third handler to your log object. You are not removing the old handlers.

You could:

    def __init__(self):
        self.handler = None
        # ... other code ...

    def activateLogFile(self, f):
        # ... other code ...
        if self.handler is not None:
            self.logger.removeHandler(self.handler)
        self.handler = logging.FileHandler(f)
        self.logger.addHandler(self.handler)

As suggested in Removing handlers from python, perhaps you could simply use:

    def activateLogFile(self, f):
        # ... other code ...
        self.logger.handlers = []
        self.logger.addHandler(handler)

bergerg · Answer 2 · 2017-08-17T21:31:37.127

What's wrong with keeping it simple?

import logging
from os.path import join, dirname, abspath

def fast_log(msg, filename='res.log'):
    logging.basicConfig(filename=filename, level=logging.INFO)
    logging.info(msg)

def do_experiment(i):
    filename = join(dirname(abspath(__file__)), "%d.log" % i)
    fast_log("experiment initialization log", filename)
    # do stuff
    fast_log("first stage", filename)
    # do a lot
    # more stuff
    # ...
    results = execute_final_stage_of_exp()
    return results

for i in range(1, 11):
    fast_log(do_experiment(i))

How to write specific outputs to multiple log files using logging in a for loop in python?

2 Answers2