Django AdminEmailHandler hangs when DB is not accesible

Question

I noticed that when my database goes down, the queries to my Django app gives timeout instead of returning 500 to the client instantaneously.

Tracing the problem I set database connect_timeout in config to 5s (explained here) and the exception appears faster on the logs but after printing the exception the client does not receive the result until 30s more.

OperationalError: (2003, "Can't connect to MySQL server on 'db.example.com' (4)")

Debugging what is happening with PDB I found that the problem is in the django.utils.log AdminEmailHandler, when it tries to generate the mail of the exception.

To generate the mail it calls django.views.debug ExceptionReporter.get_traceback_text() that looks for all frames on the traceback and then for all variables on each frame, and one of these variables is the queryset that triggered the exception.

Generating the error email makes multiple accesses to the queryset that generated the DB connection timeout, generating more DB timeouts and thus the error response to the client takes so long.

What is the best approach to avoid this problem?

score 0 · Answer 1 · answered Sep 22 '16 at 18:38

While no better alternative is given, this is the solution I have implemented.

I have setup a custom AdminEmailHandler that inspect's the exception and in case of database OperationalError skips the exception and sends and error instead.

import logging

from django.utils.log import AdminEmailHandler
from django.db.utils import OperationalError

logger = logging.getLogger('mylogger')

class CustomAdminEmailHandler(AdminEmailHandler):

        # When mail is because of exception conencting to DB, avoid rendering the email,
        # rendering email makes connections to DB making the query hang in multiple DB 
        # connection timeouts.
        if record.exc_info:
            exc_type, exc_value, exc_traceback = record.exc_info
            if exc_type == OperationalError:
                logger.error(exc_value, exc_info=False)
                return

        super(CustomAdminEmailHandler, self).emit(record)

And in settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(asctime)s %(levelname)s %(module)s:%(funcName)s() %(message)s'
        },
    },
    'handlers': {
        'send_mail': {
            'level': 'ERROR',
            'class': 'myapp.handlers.CustomdAdminEmailHandler',
            'include_html': False,
        },
    },
    'loggers': {
        'django': {
            'handlers': ['send_mail'],
            'level': 'INFO'
        },
        'mylogger': {
            'handlers': ['send_mail'],
            'level': 'INFO'
        },
    }
}

I have also set database connection timeout to 1 second, I don't want the requests to be queued in case that the database is not accessible, this may lead to a very high load when the connection returns that can get the servers down.

Django AdminEmailHandler hangs when DB is not accesible

1 Answers1