3

I'm working on a Django-based application that needs to write to a logfile from multiple processes.

I've looked at the idea from this answer which details the approach used in the multiprocessing-logging package here. Essentially, it creates a queue for handling the output to the logfile by making available a custom logging.Handler subclass, MultiProcessingHandler. The install_mp_handler function that I'm using below is part of that package and simply "wraps" the specified logger in that class.

However, I can't seem to find a way to implement it in Django in such a way that avoids throwing this error:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: /logfile.log -> /logfile.log.1

In my MyApp/settings.py file I have the handler for the logfile configured as such:

'handlers': {
        ....
        'logfile': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': BASE_DIR + "/logfile.log",
            'maxBytes': 50000,
            'backupCount': 2,
            'formatter': 'standard',
        },
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'standard'
        },
    },
    'loggers': {
        'django': {
            'handlers': ['console'],
            'propagate': True,
            'level': 'WARN',
        },
        'django.db.backends': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'MyApp': {
            'handlers': ['console', 'logfile'],
            'level': 'DEBUG',
        },

In the part of the application that runs multiple processes, I have the logging configured as such:

# Ref: https://github.com/jruere/multiprocessing-logging
import logging
from multiprocessing_logging import install_mp_handler
LOG = logging.getLogger('MyApp')
install_mp_handler()

...

if __name__ == '__main__':

...

with Pool() as pool:
    pool.map(func, iterable)

Notes: This logs to the console fine but seems to have no effect on Queuing the output to the logfile.

alphazwest
  • 3,483
  • 1
  • 27
  • 39

0 Answers0