5

I want all warnings to be handled by the way I want logs to be handled. Even these emitted during importing libraries.

This means the configuration of the logging must be done before importing libraries.

After searching some time I think configuring logging in a custom sitecustomize.py could be a solution.

But sitecustomize.py is somehow black magic only few people know it, and even less people use it.

Is there a more obvious way to ensure that my logging config is done before libraries get imported?

Unfortunately there seems to be no way to configure logging via environment variables.

Update

I got no answer which is acceptable according to my point of view. I think you need to split logging into two areas of responsibility:

  • Environment: set up logging: Which format, which files ...
  • Code: use logging. Do emit info, warn and error messages.

As soon as the first line of code gets executed by the python interpreter, the area "environment" has given the responsibility to the code. Now it is too late to do any configuration. I would like to see setting up the logging to be a part of calling the python interpreter.

One solution could be an environment variable:

PYTHON_LOGGING_CONFIG=/path-to-env/etc/config.yaml

or an argument to the interpreter:

python --logging-config=path-to-env/etc/config.yaml script.py
guettli
  • 25,042
  • 81
  • 346
  • 663
  • 3
    I understand your question, but I don't get it. What is the problem of having a library and the first thing you do in the `__init__.py` is configuring the logging using a file or other method?, as I suggested [here](http://stackoverflow.com/questions/33807362/logging-in-a-framework/33807502#33807502) – rll Nov 28 '15 at 14:59
  • This question has advice for running Python programs through a wrapper script that sets up debugging aids etc. http://stackoverflow.com/questions/32184440/making-python-run-a-few-lines-before-my-script – fpbhb Nov 28 '15 at 17:09
  • 2
    `sitecustomize` is just another module being imported. And what's wrong with the first thing your script does is setting up logging before importing stuff? Imports do not *have* to be the first thing a script does. – Martijn Pieters Nov 29 '15 at 09:42
  • I don't like it if the order of imports is important, or if there is code between the import statements at the top of a file. I like cleaning up my code with automated tools and this can break stuff. The automatic import cleaning could put imports before the logging-setup-stuff. Then again, warnings emitted during important would be emitted before the logging was set up. Next why I don't like it: if the script gets imported (but not executed) the logging gets set up again. It is easy to make the logging-setup-method idempotent, but it is all dancing around an unresolved underlying problem. – guettli Nov 30 '15 at 13:00

2 Answers2

5

There is apparently no way around explicitly configuring logging before importing anything else if you want logging to cover the imports.

However, if you want logging configuration to be more obvious, there are various tricks that will help you do that. For example, your main script could be as short as this, just to emphesize the special case:

import siteconfiguration
siteconfiguration.configure_site()  # This must be done before any other import

import application


if __name__ == "__main__":
    application.run()

That still breaks the rule of having all imports at the top, but it also clearly states why that is done. Everyone working with that code will understand your intentions.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

(The Zen of Python, by Tim Peters)

zvone
  • 18,045
  • 3
  • 49
  • 77
0

explicit import machinery

I don't like it if the order of imports is important, or if there is code between the import statements at the top of a file. I like cleaning up my code with automated tools and this can break stuff.

If you want something to happen first, then execute it first. If you want to modify how import work, you would need to do it before import is called. If your problem is caused by your editor, then either change it, find how not to reorder the clauses, or put the code in a module whose name comes first, e.g. import aaaaaa_logging_overrides.

Say your code looks like the solution @zvone has; and your editor reorders the imports from:

import siteconfiguration
siteconfiguration.configure_site()  # This must be done before any other import

import application

to 

import application
import siteconfiguration
siteconfiguration.configure_site()  # This must be done before any other import

I would probable change editors. This is very similar to having it screw up the code below, where it is obvious an editor should not reorder imports and put them before path manipulation:

import sys
sys.path.append('/nfs/some/corp/dir')
import corp.module

using sitecustomize

Your python stuff is located under sys.prefix. There should be:

  • lib/python-*/site.py
  • lib/python-*/site-packages
  • lib/python-*/sitecustomize.py (optional, depends if your admin used one)

e.g.

(test)/tmp/test > (4) cat lib/python2.7/sitecustomize.py
print 'imported sitecustomize'
(test)/tmp/test > (4) cat foo.py 
print 'executing foo'

running the example:

(test)/tmp/test > (4) python foo.py 
imported sitecustomize
executing foo

As you said, this is black magic; as a user running a script may not expect some other random code (read: possibly not under their control, and not part of the standard distribution) to run and alter the behavior.

dnozay
  • 23,846
  • 6
  • 82
  • 104