35

Possible Duplicate:
Python memory profiler

I've got a fairly complex (about 20,000) line Python program which after some development has started consuming increasing amounts of memory when it runs. What are the best tools and techniques for finding out what all the memory is being used for?

Usually this comes down to either unexpectedly keeping references to objects, or extension module bugs (which isn't particularly likely since we've been using the Python 2.4 installation for a while).

We are using various third party libraries such as Twisted, Twisted Conch and MySQLdb.

Community
  • 1
  • 1
Dickon Reed
  • 3,575
  • 4
  • 23
  • 25

2 Answers2

19

Generally, failing to close cursors is one of the most common kinds of memory leaks. The garbage collector can't see the MySQL resources involved in the cursor. MySQL doesn't know that the Python side was released unless the close() method is called explicitly.

Rule of thumb. Open, use and close cursors in as short a span of code as you can manage.

S.Lott
  • 384,516
  • 81
  • 508
  • 779
  • I have been trying to find why my system won't release resources in a program, and this was exactly it. Such a small thing, but when doing thousands of DB queries, it gets big fast! – Shakesbeery Dec 22 '14 at 20:07
  • 1
    Sounds like you should be using a with statement and an RAII-like class so you can't forget to close, even if exceptions are thrown. – Jesse Pepper Oct 17 '17 at 07:33
-2

Python's memory is managed by a garbage collector. In general, there shouldn't be a problem with memory leaking (definitely not for Python2.5 and above), unless you happen to be writing extension modules in C/C++. In that case, Valgrind (Blog post -http://bruynooghe.blogspot.com/2008/12/finding-memory-leaks-in-python.html) might be helpful. I found that this person - http://mg.pov.lt/blog/hunting-python-memleaks has used PDB and matplotlib to trace a memory leak. I hope this helps, I have no experience fixing Python memory leaks.

batbrat
  • 5,155
  • 3
  • 32
  • 38
  • 20
    In a complex program, sometimes you'll keep a reference to an object without realising it. This makes the garbage collector unable to reclaim the memory. The way the question that Sebastjan linked to is better; what I really want is to see where my memory is being used. – Dickon Reed Feb 10 '09 at 14:17