3

I have a python program that processes image frames with Python 2.7, PIL, OpenCV, and numpy/scipy. To the best of my knowledge, it does not maintain any lists of previous frame. Nevertheless, memory consumption increases steadily as the program processes more and more frames.

There are several good discussions of memory profiling solutions for Python, but they seem to focus on 32-bit or Linux solutions. What should I use with 64-bit Python 2.7 on Windows? Initial investigations suggest that the issue is with a C library. I am particularly interested in tools to help detect C library leaks or experience finding leaks in Python / OpenCV / PIL.

Community
  • 1
  • 1
Carl F.
  • 6,718
  • 3
  • 28
  • 41
  • 1
    I'm hoping there are more potential tools out there. Has anyone else struggled with this? +50 points for a good solution for dll memory leak detection. – Carl F. Apr 30 '11 at 00:53
  • 1
    There is not a lot of action on this question. I'm starting to wonder if it's a reflection of poor development tools in Windows -- especially 64-bit windows. – Carl F. May 01 '11 at 13:29
  • Thanks to all who tried to help. I don't have a solution yet, which I attribute to a lack of decent development tools for the Windows environment. I wish I could split the bounty because I got several partial answers. I've tried to award points proportionally to each contribution. – Carl F. May 06 '11 at 02:31

3 Answers3

1

I've found the tools discussed here very helpful: http://mg.pov.lt/blog/hunting-python-memleaks.html

There is a version of his code here with some additions for measure numpy array sizes.

  • Thanks, but it doesn't appear to see my memory leak. Memory consumption keeps rising and objgraph.show_largest_types(10) doesn't change. At least it works on Win64 though! – Carl F. Apr 23 '11 at 02:35
  • I'm not that familiar with OpenCV, but perhaps it's not detecting the memory used in OpenCV at the right size. If the list of python objects doesn't grow at all from frame to frame, the leak is probably in the C code or the python-to-C link –  Apr 23 '11 at 05:21
  • BTW, do you have a small test case that reproduces the problem? –  Apr 23 '11 at 05:21
  • I suspect you're right. PIL is also C code under the hood, right? The code is pretty lengthy and could be ITAR controlled so I'm afraid I can't post it. I'll edit my question to focus on detecting C leaks. – Carl F. Apr 23 '11 at 12:03
1

I had a similar sort of problem tracking down a severe memory leak in a numpy/scipy heavy code where none of the usual Python memory management tools and diagnostics detected the leak or hinted at its source.

In my case, the source of the leak was scipy interface code to the UMFPACK solver package, which was calling a C language initialization routine at every call of the interface object constructor, but never calling the de-initialization routine when the interface object was destroyed, resulting in scratch space and internal allocations leaking at the rate of about 15Mb a call. In an application with 10-20k calls, the impact was severe. Because the memory allocation was not done via the Python memory manager, things like heapy could not detect the leak.

I wound up having to use valgrind + "printf" style debugging to track down the culprit. You might need to look at non-python memory use analyzers and instrumentation tools to find out where the leak is coming from. I don't work in the Windows environment, and am not familiar with the standard toolchains, so I can't really suggest what to use. Perhaps someone else could chip in with some suggestions.

talonmies
  • 70,661
  • 34
  • 192
  • 269
1

David Malcom did a talk this year at PyCon 2011, called Dude, Where's My RAM?. He talks about debugging memory usage in Python as well as showing a tool he developed for analyzing memory usage called gdb-heap which can track memory usage down to individual bytes. Really, really great talk. I would guess it'd be difficult to use gdb-heap on Windows (maybe useful to test on another platform and possibly debug there?), but the talk covers a lot of the common issues, resolutions, etc.

Zach Kelling
  • 52,505
  • 13
  • 109
  • 108