I'm working on a Django-based application that needs to write to a logfile from multiple processes.
I've looked at the idea from this answer which details the approach used in the multiprocessing-logging
package here. Essentially, it creates a queue for handling the output to the logfile
by making available a custom logging.Handler
subclass, MultiProcessingHandler
. The install_mp_handler
function that I'm using below is part of that package and simply "wraps" the specified logger in that class.
However, I can't seem to find a way to implement it in Django in such a way that avoids throwing this error:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: /logfile.log -> /logfile.log.1
In my MyApp/settings.py
file I have the handler for the logfile configured as such:
'handlers': {
....
'logfile': {
'level': 'DEBUG',
'class': 'logging.handlers.RotatingFileHandler',
'filename': BASE_DIR + "/logfile.log",
'maxBytes': 50000,
'backupCount': 2,
'formatter': 'standard',
},
'console': {
'level': 'INFO',
'class': 'logging.StreamHandler',
'formatter': 'standard'
},
},
'loggers': {
'django': {
'handlers': ['console'],
'propagate': True,
'level': 'WARN',
},
'django.db.backends': {
'handlers': ['console'],
'level': 'DEBUG',
'propagate': False,
},
'MyApp': {
'handlers': ['console', 'logfile'],
'level': 'DEBUG',
},
In the part of the application that runs multiple processes, I have the logging configured as such:
# Ref: https://github.com/jruere/multiprocessing-logging
import logging
from multiprocessing_logging import install_mp_handler
LOG = logging.getLogger('MyApp')
install_mp_handler()
...
if __name__ == '__main__':
...
with Pool() as pool:
pool.map(func, iterable)
Notes: This logs to the console fine but seems to have no effect on Queuing the output to the logfile.