Profiling Django

Question

My django application has become painfully slow on the production. Probably it is due to some complex or unindexed queries.

Is there any django-ish way to profile my application?

Answer 1

Try the Django Debug Toolbar. It will show you what queries are executed on each page and how much time they take. It's a really useful, powerful and easy to use tool.

Also, read recommendations about Django performance in Database access optimization from the documentation.

And Django performance tips by Jacob Kaplan-Moss.

Answer 2

Just type "django-profiling" on google, you'll get these links (and more):

http://code.djangoproject.com/wiki/ProfilingDjango

http://code.google.com/p/django-profiling/

http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/

Personally I'm using the middleware approach - i.e. each user can toggle a "profiling" flag stored in a session, and if my profiling middleware notices that a flag has been set, it uses Python's hotshot module like this:

def process_view(self, request, view_func, view_args, view_kwargs):

     # setup things here, along with: settings.DEBUG=True 
     # to get a SQL dump in connection.queries

     profiler = hotshot.Profile(fname)
     response = profiler.runcall(view_func, request, *view_args, **view_kwargs)
     profiler.close()

     # process results

     return response

EDIT: For profiling SQL queries http://github.com/robhudson/django-debug-toolbar mentioned by Konstantin is a nice thing - but if your queries are really slow (probably because there are hundreds or thousands of them), then you'll be waiting insane amount of time until it gets loaded into a browser - and then it'll be hard to browse due to slowness. Also, django-debug-toolbar is by design unable to give useful insight into the internals of AJAX requests.

EDIT2: django-extensions has a great profiling command built in:

https://github.com/django-extensions/django-extensions/blob/master/docs/runprofileserver.rst

Just do this and voila:

$ mkdir /tmp/my-profile-data
$ ./manage.py runprofileserver --kcachegrind --prof-path=/tmp/my-profile-data

Answer 3

Shameless plug here, but I recently made https://github.com/django-silk/silk for this purpose. It's somewhat similar to django toolbar but with history, code profiling and more fine grained control over everything.

Answer 4

For profiling data access (which is where the bottleneck is most of the time) check out django-live-profiler. Unlike Django Debug Toolbar it collects data across all requests simultaneously and you can run it in production without too much performance overhead or exposing your app internals.

Answer 5

I needed to profile a Django app recently and tried many of these suggestions. I ended up using pyinstrument instead, which can be added to a Django app using a single update to the middleware list and provides a stack-based view of the timings.

Quick summary of my experience with some other tools:

• Django Debug Toolbar is great if you the issue is due to SQL queries and works well in combination with pyinstrument
• django-silk works well, but requires adding a context manager or decorator to each part of the stack where you want sub-request timings. It also provides an easy way to access cProfile timings and automatically displays ajax timings, both of which can be really helpful.
• djdt-flamegraph looked promising, but the page never actually rendered on my system.

Compared to the other tools I tried, pyinstrument was dramatically easier to install and to use.

Answer 6

For all you KCacheGrind fans, I find it's very easy to use the shell in tandem with Django's fantastic test Client for generating profile logs on-the-fly, especially in production. I've used this technique now on several occasions because it has a light touch — no pesky middleware or third-party Django applications are required!

For example, to profile a particular view that seems to be running slow, you could crack open the shell and type this code:

from django.test import Client
import hotshot

c = Client()
profiler = hotshot.Profile("yourprofile.prof")  # saves a logfile to your pwd
profiler.runcall(c.get, "/pattern/matching/your/view/")
profiler.close()

To visualize the resulting log, I've used hotshot2cachegrind:

• http://kcachegrind.sourceforge.net/html/ContribPython.html

But there are other options as well:

• http://www.vrplumber.com/programming/runsnakerun/
• https://code.djangoproject.com/wiki/ProfilingDjango

Answer 7

When the views are not HTML, for example JSON, use simple middleware methods for profiling.

Here are a couple examples:

https://gist.github.com/1229685 - capture all sql calls went into the view

https://gist.github.com/1229681 - profile all method calls used to create the view

Answer 8

You can use line_profiler.

It allows to display a line-by-line analysis of your code with the time alongside of each line (When a line is hit several times, the time is summed up also).

It's used with not-Django python code but there's a little trick to use it on Django in fact: https://stackoverflow.com/a/68163807/1937033

Answer 9

I am using silk for live profiling and inspection of Django application. This is a great tool. You can have a look on it.

https://github.com/jazzband/django-silk