1

I've got a long-running Python 3 task on a Linux server that is being Killed, which is apparently due to resource exhaustion. The task reads in a data set of about 5000 items, runs a for loop over the list, and does some rather intensive processing on each element. It got killed before finishing the first 100. The processing data is not interdependent from one iteration of the loop to another, so theoretically everything should be garbage collectable between iterations, but somehow it's still getting killed for running out of memory.

Are there any useful techniques to get a memory profile of a Python 3 script and figure out where it's leaking memory, so I can debug this? Most of the tools I've looked up, such as the memory_profiler library, seem to need you to already know where the problem is before it will help you find it, requiring you to annotate a known-bad function with a specific decorator. The problem is, this is a pretty big program, with over 40 .py scripts and a dozen 3rd party libraries, some of which are native code (numpy, scipy, etc.) and I don't even know where to start looking.

What's the best way to go about figuring out where the leaks exist?

Community
  • 1
  • 1
Mason Wheeler
  • 82,511
  • 50
  • 270
  • 477
  • This might be a dupe of (the second part of) http://stackoverflow.com/questions/1435415/python-memory-leaks Many years have passed, so new tools may be available or the tools mentioned over there might have been abandoned. The changing nature of reality over time is not necessarily well-served by SO's anti-dupe policy, but the ideal is that if the answer has changed then that other question is the place to put the new answer. – Steve Jessop Jun 06 '16 at 12:59

0 Answers0