Django - logging each view's action

Question

I'm thinking of creating a log system for my django web application. The web application is quite comprehensive in its use (covers all aspects of a business's processes) so I'd like to track every event that happens. Specifically, I'd like to log every view that runs and not just the "main" ones and, potentially, log what is happening within the view as its executed.

While I'm in the "idea" stage of the logging system, I've quickly hit a few questions that leave me unsure how to proceed. Here are the main questions I have:

• I'm thinking of logging all of the events in the same MySQL database that the main web app holds its data. The concern I have is bloating the MySQL database into a massive DB. Also, if the DB crashes or is destroyed somehow (yes I have backups) I'll loose my log too which blows away any ability to track down the problem. Do I use a seperate DB or just go with text files?
• How granular do I go? Initially I was thinking of simply logging things like, "Date - In view myView". However, as I'm thinking about it, it would be nice to log all the stuff that happens within the view. Doing this could make the log massive! and would also make my code ugly with so many log entry lines mixed into the code. This kind of detail:
  • Date - entered view myView
  • Date - in view myView, retrieved object myObject from the DB
  • Date - in view myView, setting myObject field myField to myNewValue
  • Date - leaving myView

Those are my main thoughts at this point. Any advice on this front?

Thanks

Answer 1

I think the best and right way is to create your own custom middleware where you can log actually everything you need.

Here are some links on the subject:

• middleware snippets
  • http://djangosnippets.org/snippets/2624/
  • http://djangosnippets.org/snippets/290/
  • http://djangosnippets.org/snippets/264/
• django-logging-middleware (pretty old by may give you an idea)
• django-request
• django.db.backends logging
• Is there a Django middleware/plugin that logs all my requests in a organized fashion?
• Django verbose request logging
• log all sql queries
• django orm, how to view (or log) the executed query?

Also, consider using sentry error logging and aggregation platform instead of writing logs into the database. FYI, see using a database for logging.

Answer 2

If you want to log any action run in every view, you can for example, replace entered view A and exited view A by a line in these words: view A - 147ms. As alecxe stated, you can log requests/SQL, there are plenty of ways to do it with middleware. About database (object) actions, you can tie individual saves, updates and deletes with signals. For bulk updates and deletes, you could (it's not a clean way but it would work) monkey-patch manager and queryset methods to add logging. This way you can log actions rather than SQL. I would see lines like this:

[2013/09/11 15:11:12.0153] view   app.module.view       200  148ms
[2013/09/11 15:11:12.0189] orm    save:auth.User,id=1        3ms

This is a quick and dirty proposal, but, maybe it's worth it.