0

I'm definitely missing something when it comes to logging using the logging module. I can get it to work just fine if I have just 1 file with everything in it, but when I try to split things up into separate files or packages I'm not sure what to do.

Suppose I have a package with the following directory tree:

mypackage/
├─ __init__.py
├─ utils.py
├─ my_logger.py
main.py

In my package I have a logger (in my_logger.py) that is configured nearly to my liking. In my main.py file, I initialize that logger and add a FileHandler to start writing to a file.

However, I'd also like to log to this file from other files in my package. Say, for example, the file utils.py. How do I make the instance of logger in utils.py know about the FileHandler?

Suppose I have a series of functions in utils.py, say, myfunc1() and myfunc2(), and I'd like each to also log to the same file. I don't want to have to pass my instance of logger to each individual function.

There feels like an obvious setting I am missing. What should I have in which file in order to make sure they all know about the same logger instance everywhere in the package?

(A detail that may or may not be relevant — I've actually subclassed Logger to give it some custom behaviour. But this shouldn't affect the answer to this question, I don't think.)

Drphoton
  • 164
  • 9
  • The general idea for loggers is that the logger for individual modules should simply be set up using [`getLogger(__name__)`](https://stackoverflow.com/questions/15727420/using-logging-in-multiple-modules) and if you want specific functions for logging when executing your main program, that's the only location where the custom configurations be used (e.g. attaching a handler to log to specific file(s)). Another [thread](https://stackoverflow.com/questions/50714316/how-to-use-logging-getlogger-name-in-multiple-modules) describing another similar setup. – metatoaster Jan 06 '23 at 01:17
  • So each file (module) in the file should make their own "getLogger" call to log to? What if I want to consolidate all the results from all the different files into a single log file? – Drphoton Jan 06 '23 at 01:19
  • They are not logged to different files - modules shouldn't do anything to the logger instance they created using `getLogger(__name__)` (note that's the only setup command, so they won't have handlers set up to write to their own files), they should leave it to the main program to set it up. It would be easier for you to actually try it out and see the results yourself. – metatoaster Jan 06 '23 at 01:20
  • So the loggers each just log to a void, but in the main program we specify what file to log to? But then how does it know? I need an example... I'm going to read your link and see if it solves my problem. – Drphoton Jan 06 '23 at 01:22
  • 1
    They are _not_ logged to a void, they are logged to the named logger as per the `__name__` of the module, e.g. calling `getLogger(__name__)` inside `mypackage/utils.py` should return a logger named `'mypackage.utils'`, and logs will be written to it. In your main application you should get the root logger just by `getLogger()` and attach handlers to it, which will capture all the subloggers. If you want to only deal with messages in your package, simply `getLogger('mypackage')` which will capture all loggers underneath `'mypackage.*'`, which includes `mypackage.utils`. – metatoaster Jan 06 '23 at 01:23
  • I see. So somehow wherever you call `getLogger()` it has access to the loggers everywhere in the namespace? As in, `getLogger('a')` will produce the same logger wherever it is called within the interpreter? – Drphoton Jan 06 '23 at 01:30
  • 1
    Yes. `getLogger('a') == getLogger('a')` is `True`. Logging to a logger returned by `getLogger('a.b')` will show up for handlers attached to `getLogger('a')`. – metatoaster Jan 06 '23 at 01:35
  • Thanks so much @metatoaster! I never knew this. I'm going to test this with toy packages here and I'll put it as the answer if it solves my problem. – Drphoton Jan 06 '23 at 01:41
  • @metatoaster please see the answer I posted. This seems to work great! – Drphoton Jan 06 '23 at 15:50

1 Answers1

1

Thanks to @metatoaster I have the solution I was looking for.

I have the files organized in the tree above.

In mypackage.my_logger.py:

import logging
import sys

logger = logging.getLogger('mypackage')
ch = logging.StreamHandler(sys.stdout)
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)

In mypackage.utils.py:

import logging

logger = logging.getLogger('mypackage')

logger.info('Running in the body of mypackage.utils.py')


def myfunc():
    logger.info("Running in myfunc() in mypackage.utils.py")

In my main.py file:

from mypackage.my_logger import logging

from mypackage.utils import myfunc


logger = logging.getLogger('mypackage')

logger.info('Running in main.py')

myfunc()

The output is:

Running in the body of mypackage.utils.py
Running in main.py
Running in func() in mypackage.utils.py

Note how, by importing logging from mypackage.my_logger, I'm running the setup for the StreamHandler in the body of that module.

I understand now why the convention is to always get a logger that calls for __name__, as in getLogger(__name__). Explicitly naming the package everywhere works for my current usecase. Thanks again for the help!

Drphoton
  • 164
  • 9
  • 1
    Yeah that's certainly an approach if you want to isolate your logging setup to a module that gets automatically set up if you import it, though my recommendation is to wrap that in a function to avoid side-effects - this prevents people who don't want random log files being generated just simply by importing your package being inconvenienced by what they deem as unwanted output. Your `main` program will import the function (instead of logging) and call it to set up the logger only upon running. Also reason for using the `__name__` helps identify the module that logged the message. – metatoaster Jan 06 '23 at 23:26