Django: Logging from batch/cron scripts

Question

Up to now we run cronjobs like this:

@daily do_daily_magic > log/do_daily_magic.log

Up to now we don't use the logging config of django:

https://docs.djangoproject.com/en/1.8/topics/logging/

But I would like to change to the default django way.

How to write a batch script for django which outputs all logging to stdout?

Answer 1

If you are trying to run a python script from outside of the Django environment then you need to source the Django paths to obtain the libraries and methods leveraged from Django. This can be done, but it is kind-of dirty.

Your python script needs to source information form Django, e.g.:

import sys, os
sys.path.append('/path/to/your/django/app')
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.conf import settings

You may want to follow this : http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/

It is more pythonic to write the log or output form the python code instead of relying upon stdout. If you follow that approach, then your cron entry should be

@daily do_daily_magic 2>&1 | logger

which will take all STDOUT and STDERR and send it to logger to get captured by syslog (e.g /var/log/sys.log). Since your python code will be directing output to it's own file the only thin that will come out of the cron entry will be debug message or errors that are not directed to the log, including problems with cron, permissions, exceptions, etc. and be rotated properly)

Answer 2

See console handler than flush data through logging.StreamHandler, also remember than default behavior for logging.StreamHandler is loggin to stderr:

If stream is specified, the instance will use it for logging output; otherwise, sys.stderr will be used.

You can write your own handler and class to loggin to sys.stdout, my_utils.py:

import logging, sys

class StreamHandlerStOut( logging.StreamHandler )
    def __init__( self ):
        super(StreamHandlerStOut, self).__init__(sys.stderr) 

And set your new class to your handler on settings.py:

import os
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'my_console': {
            #'class': 'logging.StreamHandler',
            'class': 'my_utils.StreamHandlerStOut',
        },
    },
    'loggers': {
        'my_logger': {
            'handlers': ['my_console'],
            'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
        },
    },
}

Remember to use your logger on your cron scripts code.

Answer 3

I had a similar use case and handled it by using a custom management command. In my opinion, it's much more elegant than a standalone python script since everything is setup automatically and I can always manually run the command through manage.py.

You can do this by putting your logic in a module at

your_app/management/commands/your_daily_magic_command.py

Don't forget to also create (if it doesn't already exist):

your_app/management/__init__.py
your_app/management/commands/__init__.py

Inside your_daily_magic_command.py, it should look something like this:

from django.core.management.base import BaseCommand, CommandError
##import logging
##logger = logging.getLogger(__name__)


class Command(BaseCommand):
    #args = '<required_arg1> [optional_arg2]'
    help = 'Put your help text here to be shown in manage.py help'

    def handle(self, *args, **options):
        #if len(args) == 0:
        #    raise CommandError('Not enough arguments. Please see help')
        #if len(args) > 2:
        #    raise CommandError('Too many arguments. Please see help')

        self.stdout.write('Starting my daily magic stuff...')
        #self.stdout.write('Arguments: %s' % args)

        # Your daily magic logic here

        self.stdout.write('Finished my daily magic stuff :)')
        ##logger.debug("A debug message...")
        ##logger.info("...using django's logging mechanism")

A nice thing is it'll also handle command line arguments for you. In my example, the single commented lines show you how to use one required and one optional arguments so you can just edit it from there.

There's no need to use the logging package as you can simply use self.stdout.write (see code)

For official examples, see django.core.mangement.commands.*

To run your command:

python /path/to/your/project/manage.py your_daily_magic_command

---

If you really want to use the python logging package with django's shortcuts, you will need to set up your settings.LOGGING. Here's an example:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            # You can change your format here
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        },
    },
    'handlers': {
        'console_out': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'standard',
            # 'strm' in Python 2.6. See http://codeinthehole.com/writing/console-logging-to-stdout-in-django/
            'stream': sys.stdout,
        },
        'console_err': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'standard',
        }
    },
    'loggers': {
        'your_app_path.management.commands.your_daily_magic_command': {
            'handlers': ['console_out'],
            'level': 'DEBUG',  # DEBUG level is lower than INFO
            'propagate': True,
        },
        # Default logger: http://stackoverflow.com/questions/5438642/django-setup-default-logging
        '': {
            # For special handling, see http://code.activestate.com/recipes/576819-logging-to-console-without-surprises/
            'handlers': ['console_err'],
            'level': 'INFO',   # Show only INFO and up
            'propagate': True,
        },
    }
}

In the your_daily_magic_command.py example above, the double commented (##) lines show you how to use this.

The logger name can be anything corresponding to your settings.LOGGING['loggers'] entry, but using __name__ to refer to the current module is standard practice.

Answer 4

Define logging in settings.py in following way:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/path/to/do_daily_magic.log',
        },
    },
    'loggers': {
        'myapp.myscript.logger': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Than you can use logger myapp.myscript.logger to write info to the file directly from django.

import logging

logger = logging.getLogger('myapp.myscript.logger')


class Command(BaseCommand):
    def handle(self, *args, **options):
        # do some stuff
        logger.info("Some info for logs")

However, I suggest you to keep logging from crontab as well. Because it can handle any error that happens before invocation of Django. For example it will also handle missing python interpreter or wrong environment.

P.S. If you just want to write all logs from django to stdout use following config for logger

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        'myapp.myscript.logger': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}