How to structure the Logging in this case (python)

Question

I was wondering what would be the best way for me to structure my logs in a special situation.

I have a series of python services that use the same python files for communicating (ex. com.py) with the HW. I have logging implemented in this modules and i would like for it to be dependent(associated) with the main service that is calling the modules.

How should i structure the logger logic so that if i have:

main_service_1->module_for_comunication

The logging goes to file main_serv_1.log

main_service_2->module_for_comunication

The logging goes to file main_serv_2.log

What would be the best practice in this case without harcoding anything?

Is there a way to know the file which is importing the com.py, so that i am able inside of the com.py, to use this information to adapt the logging to the caller?

@LaurIvan Can you be more explicit please, and clarify a little bit better your suggestion? — LPS, Mar 11 '16 at 17:11
When you say service, do you mean different **modules** that import `com.py`, but are part of the same process (interpreter), or two different processes, as in two separate invocations of a Python interpreter for s1 and s2? — Lukas Graf, Mar 11 '16 at 18:58
@LukasGraf I mean two separate invocations of the python interpreter. Like calling functions foo() and bar() (foo and bar mentioned on your post down) in separate main .py files. — LPS, Mar 14 '16 at 18:34

score 1 · Accepted Answer · answered Mar 11 '16 at 18:39

In my experience, for a situation like this, the cleanest and easiest to implement strategy is to pass the logger to the code that does the logging.

So, create a logger for each service you want to have log to a different file, and pass that logger in to the code from your communications module. You can use __name__ to get the name of the current module (the actual module name, without the .py extension).

In the example below I implemented a fallback for the case when no logger is passed in as well.

com.py

from log import setup_logger


class Communicator(object):

    def __init__(self, logger=None):
        if logger is None:
            logger = setup_logger(__name__)

        self.log = logger

    def send(self, data):
        self.log.info('Sending %s bytes of data' % len(data))

svc_foo.py

from com import Communicator
from log import setup_logger


logger = setup_logger(__name__)


def foo():
    c = Communicator(logger)
    c.send('foo')

svc_bar.py

from com import Communicator
from log import setup_logger


logger = setup_logger(__name__)


def bar():
    c = Communicator(logger)
    c.send('bar')

log.py

from logging import FileHandler
import logging


def setup_logger(name):
    logger = logging.getLogger(name)
    handler = FileHandler('%s.log' % name)
    logger.addHandler(handler)
    return logger

main.py

from svc_bar import bar
from svc_foo import foo
import logging


# Add a StreamHandler for the root logger, so we get some console output in
# addition to file logging (for easy of testing). Also set the level for
# the root level to INFO so our messages don't get filtered.
logging.basicConfig(level=logging.INFO)


foo()
bar()

So, when you execute python main.py, this is what you'll get:

On the console:

INFO:svc_foo:Sending 3 bytes of data
INFO:svc_bar:Sending 3 bytes of data

And svc_foo.log and svc_bar.log each will have one line

Sending 3 bytes of data

If a client of the Communicator class uses it without passing in a logger, the log output will end up in com.log (fallback).

Here's a [gist](https://gist.github.com/lukasgraf/ffe5c41d2dcfe25bfa11) for easier downloading of the example — Lukas Graf, Mar 11 '16 at 18:50
It makes sense, if you put the example like that, but imagine the following. If instead of the class **Comunication**, you have a class **Network** (network.py) wich is similiar to Comunication and a class **Device** (devices.py), which is going to be instantiated a lot of times (example: a network and all its devices). Does it make sense to use the same logging logic you mentioned? For me it doesnt seem as logic to do it for device class as it does for the Network class. I am not sure if i am making sense... — LPS, Mar 14 '16 at 18:07

score 0 · Answer 2 · edited May 23 '17 at 12:07

0

I see several options:

Option 1

Use __file__. __file__ is the pathname of the file from which the module was loaded (doc). depending of your structure, you should be able to identify the module by performing an os.path.split() like so:

If the folder structure is

+- module1
|    +- __init__.py
|    +- main.py
+- module2
     +- __init__.py
     +- main.py

you should be able to obtain the module name with a code placed in main.py:

def get_name():
    module_name = os.path.split(__file__)[-2]
    return module_name

This is not exactly DRY because you need the same code in both main.py. Reference here.

Option 2

A bit cleaner is to open 2 terminal windows and use an environment variable. E.g. you can define MOD_LOG_NAME as MOD_LOG_NAME="main_service_1" in one terminal and MOD_LOG_NAME="main_service_2" in the other one. Then, in your python code you can use something like:

import os
LOG_PATH_NAME os.environ['MOD_LOG_NAME']

This follows separation of concerns.

Update (since the question evolved a bit)

Once you've established the distinct name, all you have to do is to configure the logger:

import logging
logging.basicConfig(filename=LOG_PATH_NAME,level=logging.DEBUG)

(or get_name())and run the program.

edited May 23 '17 at 12:07

Community

1
1

answered Mar 11 '16 at 18:19

Laur Ivan

4,117
3
38
62

How would that solve the OPs problem? The actual loggin calls, say `logger.warn()`, still happen in *one place*, in `com.py`. How would you pass that information to either the logging call or the place where file handlers are being set up? – Lukas Graf Mar 11 '16 at 18:41
You don't need to pass the information around. You only need to configure the logger (like in the updated answer). When used, depending on the source where it's ran from (or environment variable) you'll get different output files. – Laur Ivan Mar 12 '16 at 12:38
I see, you're assuming those are two completely separate invocations (processes) where the common module is being used. That obviously makes it much easier. – Lukas Graf Mar 12 '16 at 17:25
Yeah. if there's only one, you'd need to have 2 loggers and pass the relevant one as parameter where needed. – Laur Ivan Mar 12 '16 at 17:46

How to structure the Logging in this case (python)

2 Answers2