How to log source file name and line number in Python

Question

Is it possible to decorate/extend the python standard logging system, so that when a logging method is invoked it also logs the file and the line number where it was invoked or maybe the method that invoked it?

Answer 1

Sure, check formatters in logging docs. Specifically the lineno and pathname variables.

%(pathname)s Full pathname of the source file where the logging call was issued(if available).

%(filename)s Filename portion of pathname.

%(module)s Module (name portion of filename).

%(funcName)s Name of function containing the logging call.

%(lineno)d Source line number where the logging call was issued (if available).

Looks something like this:

formatter = logging.Formatter('[%(asctime)s] p%(process)s {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s','%m-%d %H:%M:%S')

Answer 2

On top of Seb's very useful answer, here is a handy code snippet that demonstrates the logger usage with a reasonable format:

#!/usr/bin/env python
import logging

logging.basicConfig(format='%(asctime)s,%(msecs)03d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
    datefmt='%Y-%m-%d:%H:%M:%S',
    level=logging.DEBUG)

logger = logging.getLogger(__name__)
logger.debug("This is a debug log")
logger.info("This is an info log")
logger.critical("This is critical")
logger.error("An error occurred")

Generates this output:

2017-06-06:17:07:02,158 DEBUG    [log.py:11] This is a debug log
2017-06-06:17:07:02,158 INFO     [log.py:12] This is an info log
2017-06-06:17:07:02,158 CRITICAL [log.py:13] This is critical
2017-06-06:17:07:02,158 ERROR    [log.py:14] An error occurred

Answer 3

import logging

# your imports above ...


logging.basicConfig(
    format='%(asctime)s,%(msecs)d %(levelname)-8s [%(pathname)s:%(lineno)d in ' \
           'function %(funcName)s] %(message)s',
    datefmt='%Y-%m-%d:%H:%M:%S',
    level=logging.DEBUG
)

logger = logging.getLogger(__name__)

# your classes and methods below ...
# A very naive sample of usage:
try:
    logger.info('Sample of info log')
    # your code here
except Exception as e:
    logger.error(e)

Different from the other answers, this will log the full path of file and the function name that might have occurred an error. This is useful if you have a project with more than one module and several files with the same name distributed in these modules.

Example output:

2022-12-02:10:00:00,000 INFO     [<stdin>:2 in function <module>] Sample of info log

2022-12-02:10:00:00,000 INFO     [<full path>/logging_test_file.py:15 in function <module>] Sample of info log

Answer 4

To build on the above in a way that sends debug logging to standard out:

import logging
import sys

root = logging.getLogger()
root.setLevel(logging.DEBUG)

ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
formatter = logging.Formatter(FORMAT)
ch.setFormatter(formatter)
root.addHandler(ch)

logging.debug("I am sent to standard out.")

Putting the above into a file called debug_logging_example.py produces the output:

[debug_logging_example.py:14 -             <module>() ] I am sent to standard out.

Then if you want to turn off logging comment out root.setLevel(logging.DEBUG).

For single files (e.g. class assignments) I've found this a far better way of doing this as opposed to using print() statements. Where it allows you to turn the debug output off in a single place before you submit it.

Answer 5

For devs using PyCharm or Eclipse pydev, the following will produce a link to the source of the log statement in the console log output:

import logging, sys, os
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(message)s | \'%(name)s:%(lineno)s\'')
log = logging.getLogger(os.path.basename(__file__))


log.debug("hello logging linked to source")

See Pydev source file hyperlinks in Eclipse console for longer discussion and history.

Answer 6

If the logger is set with the GetLogger(name) option, where the name is a name that you specified you can also format the logger using %(name)s. You can specify a different name in every file with the GetLogger function, and when a log is produced you will know from which file comes through the name you set.

Example:

import logging

logging.getLogger("main")
logging.basicConfig(#filename=log_fpath, 
                    level=log_level,
                    format='[%(asctime)s] src:%(name)s %(levelname)s:%(message)s',
                    handlers=[logging.FileHandler(log_fpath)])