Limit Python VM memory

Question

I'm trying to find a way to limit the memory available for the Python VM, as the option "-Xmx" in the Java VM does. (The idea is to be able to play with the MemoryError exception)

I'm not sure this option exist but there may be a solution using a command of the OS to "isolate" a process and its memory.

Thank you.

score 34 · Answer 1 · answered Mar 25 '11 at 13:34

34

On Linux I'm using the resource module:

import resource
resource.setrlimit(resource.RLIMIT_AS, (megs * 1048576L, -1L))

answered Mar 25 '11 at 13:34

Joril

19,961
13
71
88

score 12 · Accepted Answer · answered Nov 19 '09 at 00:09

12

On *nix you can play around with the ulimit command, specifically the -m (max memory size) and -v (virtual memory).

answered Nov 19 '09 at 00:09

mhawke

84,695
9
117
138

score 3 · Answer 3 · answered Nov 19 '09 at 00:18

3

Don't waste any time on this.

Instead, if you want to play with MemoryError exceptions create a design that isolates object construction so you can test it.

Instead of this

for i in range(1000000000000000000000):
    try:
        y = AnotherClass()
    except MemoryError:
        # the thing we wanted to test

Consider this.

for i in range(1000000000000000000000):
    try:
        y = makeAnotherClass()
    except MemoryError:
        # the thing we wanted to test

This requires one tiny addition to your design.

class AnotherClass( object ):
    def __init__( self, *args, **kw ):
    blah blah blah

def makeAnotherClass( *args, **kw ):
    return AnotherClass( *args, **kw )

The extra function -- in the long run -- proves to be a good design pattern. It's a Factory, and you often need something like it.

You can then replace this makeAnotherClass with something like this.

class Maker( object ):
    def __init__( self, count= 12 ):
        self.count= count
    def __call__( self, *args, **kw ):
        if self.count == 0:
            raise MemoryError
        self.count -= 1
        return AnotherClass( *args, **kw )
 makeAnotherClass= Maker( count=12 )

This version will raise an exception without you having to limit memory in any obscure, unsupportable, complex or magical way.

answered Nov 19 '09 at 00:18

S.Lott

384,516
81
508
779

hum, funny answer :) Definitely more complicated than the first one and intrusive in my code. Oh, and thanks for teaching us what the factory design pattern is. – Nov 19 '09 at 00:29
In your examples above, you might want to use xrange instead of range (assuming this isn't Python 3), otherwise you might run out of memory for the wrong reason. – Ned Deily Nov 19 '09 at 00:42
1

@tweksteen: if you think it's "intrusive" then you haven't done proper test-driven design. Also, if you're not using factories, you're not allowing for expansion of your class hierarchies. You should -- seriously -- rethink your design so that you use factories. – S.Lott Nov 19 '09 at 01:00
1

@Ned Deily: What is the chance that I wasn't serious about that silly-looking number? – S.Lott Nov 19 '09 at 01:01
2

@S.Lott: about 1 in 1000000000000000000000 ?? – Ned Deily Nov 19 '09 at 01:29
I realize this is an old question, but just for the record: there is such a thing as third-party code and one might not feel like digging into it in order to figure out which particular object in that code eats up all the memory. In this case all I am interested in is making sure the process doesn't stall my machine. – Qnan Sep 29 '14 at 08:21

Limit Python VM memory

3 Answers3

Linked