Using heapy to track down memory leaks in Django app

Question

I've followed excellent post here how to setup heapy with Django: http://www.toofishes.net/blog/using-guppy-debug-django-memory-leaks/

I've commanded hp.setref() and now after awhile I get also data with hp.heap():

>>> hp.heap()
Partition of a set of 12075 objects. Total size = 1515496 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0   4048  34   339656  22    339656  22 str
     1   3112  26   269368  18    609024  40 tuple
     2    171   1   169992  11    779016  51 dict (no owner)
     3   1207  10   144440  10    923456  61 list
     4     49   0   102040   7   1025496  68 dict of module
     5    591   5    66984   4   1092480  72 unicode
     6    498   4    59760   4   1152240  76 function
     7    433   4    51960   3   1204200  79 types.CodeType
     8     57   0    50480   3   1254680  83 type
     9     36   0    31584   2   1286264  85 dict of class

But what now? What I should understand from this output? How I should start tracking down where those 'str' and 'tuple' objects belong?

With get_rp, I get following output:

>>> hp.heap().get_rp()
Reference Pattern by <[dict of] class>.
 0: _ --- [-] 12000 (0xd1d340 | 0xd88b50 | 0xf63f00 | __builtin__.Struct | __...
 1: a      [-] 137 dict (no owner): 0x761c30*160, 0x7655d0*1491, 0x781640*9...
 2: aa ---- [-] 45 dict of django.db.models.options.Options: 0xcf3110...
 3: a3       [-] 45 django.db.models.options.Options: 0xcf3110, 0xf0bb10...
 4: a4 ------ [-] 140 dict of django.db.models.related.RelatedObject: 0x10bec...
 5: a5         [-] 140 django.db.models.related.RelatedObject: 0xf14450...
 6: a6 -------- [-] 63 dict of django.db.models.fields.related.ForeignKey: 0x...
 7: a7           [+] 63 django.db.models.fields.related.ForeignKey: 0xf0e690...
 8: a5b ------- [-] 7 dict of django.db.models.fields.related.OneToOneField: ...
 9: a5ba         [+] 7 django.db.models.fields.related.OneToOneField: 0x15447...

Is the correct assumption now that it's Django that is leaking memory? But what are those dict's who don't have owner?

Ned Batchelder · Answer 1 · 2011-08-07T12:43:03.020

0

I don't have any experience with heapy, but in my experience, Django (and most other Python programs) don't leak memory, but they also don't clean up memory as pristinely as some would like.

Also, Django has settings that cause it to consume memory for diagnostic reasons. For example, setting DEBUG=True can cause it to hold on to all SQL queries, so the longer the process runs, the more memory it uses.

UPDATE: Your problem isn't in your Python code. Look at the summary heapy is giving you: the total size of memory represented there is 1.5Mb! When Python programs truly leak, the most common cause is a leaky C extension. Do you have any C extensions that you are running under your Django process?

edited Aug 07 '11 at 12:43

answered Aug 07 '11 at 12:16

Ned Batchelder

364,293
75
561
662

2

Thanks but that's not really answer to my question. It was about how to interpret the results. We clearly have memory leak (processes eating 2GB or mem) and it's not related to DEBUG=True as it's False in our production environment. – petteri Aug 07 '11 at 12:26
With couple of hundred of thousands model instances you can consume large memory portion without having a leak. – Davor Lucic Aug 07 '11 at 13:17
hmm how you can make assumption that it's _only_ 1,5MB without knowing how long the code has run? But anyway, yes it's possible that it's also native code that is leaking, that's what I'm trying to solve :) (note that the situation is after setting a reference) – petteri Aug 07 '11 at 13:31
I did make an assumption, true: I assumed you were measuring the condition you cared about! :-) Use heapy after your code gets to 2GB, the 1.5Mb situation may be nothing like your real problem. – Ned Batchelder Aug 07 '11 at 13:35

Using heapy to track down memory leaks in Django app

1 Answers1

Linked