4

I have a system with 16GB of memory. I run a python script for some data mining application and the process takes up the entire 16GB. I want to limit the python process to take up only a limited amount of memory.

Is it possible to do this? If yes, how?

Update:

The application retrieves hotel review data from a huge database and tries to build graphs among hotels and users for some analysis. The data structure to hold the data goes beyond 16GB.

bdhar
  • 21,619
  • 17
  • 70
  • 86
  • Could you possibly post some example code so that we can be of more use? – Slater Victoroff Jul 11 '12 at 21:56
  • On *nix, use `ulimit` (see http://stackoverflow.com/questions/1760025/limit-python-vm-memory) – hcarver Jul 11 '12 at 21:56
  • 4
    There are various ways to restrict the amount of total memory allowed to be used (e.g., `ulimit`), but this will cause the application to fail when it reaches the limit. It is a design flaw of the application if it is taking up 16GB of memory, and you haven't given us any details of the application, and therefore we don't have enough information to help you solve it. – Jay Sullivan Jul 11 '12 at 21:58
  • You could check something like this: http://www.lshift.net/blog/2008/11/14/tracing-python-memory-leaks – user1277476 Jul 11 '12 at 23:13
  • @notfed: I have updated the question with more detail on the application! Thanks. – bdhar Jul 14 '12 at 17:25

1 Answers1

2

You have an application which has a data structure that appears to require 16GB+ of RAM, right? If so, then limiting its RAM will cause the application to fail, will it not?

Is there a reason all of this data needs to be in RAM all at once? Likely, the application could be redesigned to take up less than 16GB of RAM at a time. For example, searching all file contents of all files on a computer does not require having all files open in RAM all at once, but only one at a time.

My point is that "limiting the RAM usage" isn't likely to be solvable any way other than redesigning the application.

Jay Sullivan
  • 17,332
  • 11
  • 62
  • 86
  • It *could* be that his algorithm is hard to bound space-wise, and he just wants the process killed before his computer starts hitting swap. In that case, he should look at `ulimit`, as has been mentioned. – gspr Jul 17 '12 at 15:17