11

In a Python script, I want to set a memory limit for a certain function call. I looked at how to limit heap size; however, I don't want to limit the memory of the entire running Python process -- i.e. setting the memory limit before and after the function call.

Is there any way to make a function call with a given amount of memory, so that the memory limit doesn't affect the caller?

trincot
  • 317,000
  • 35
  • 244
  • 286
Zach
  • 998
  • 1
  • 9
  • 26
  • 9
    Open a new process and limit the heap size. – JBernardo Apr 22 '12 at 17:00
  • 2
    Or, in theory you can also inspect the current frame and recursively build a tree with all objects in it and estimate the memory used, but it ain't simple. Check how memory profilers like guppy.heapy do it. – Pedro Werneck Apr 22 '12 at 17:27
  • 5
    But the real question is, why do you want to do that? There's no builtin feature for this, so maybe you are approaching the problem in the wrong way. – Pedro Werneck Apr 22 '12 at 17:28
  • 1
    I am building a Python sandbox and I would like to set a memory limit lest the users submit code that drains the server's memory. – Zach Apr 23 '12 at 05:01
  • Unless you're very strict about what you accept, the user can do all sorts of nasty things, even in a generally well-behaved language such as Python. I think memory usage isn't the biggest worry in such a scenario... – cvoinescu Apr 24 '12 at 23:01
  • This is but one of the things I'm trying to tie down for the sandbox. At the same time, I'm looking how to prevent I/O, sockets, or general introspection workarounds. – Zach Apr 25 '12 at 00:59
  • so you search for a complete sandboxing solution. http://doc.pypy.org/en/latest/sandbox.html is the only secure python sandbox I am aware of, and it uses a different approach. – ch3ka Apr 25 '12 at 11:57
  • I don't think this counts as a real answer, but check these links for an approach that might be adapted for what you want (but counting process memory): http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/ https://news.ycombinator.com/item?id=3887622 – TryPyPy Apr 26 '12 at 00:49

1 Answers1

2

No, this is impossible. Python doesn't keep track of which functions are responsible for allocating new memory, and nor does libc, so there's no way to limit memory usage just by a single function.

In order to do this you would have to modify Python so that you use a new function for allocating memory that requires it to be specified what Python function is responsible, so that it can reject the memory allocation if that function goes over its limit.

The only other way to do this would indeed be to execute the function in a separate process with limited memory, as @JBernardo said.

As for implementing sandboxes, there already are relatively well tested implementations. Is there any reason you can't use those? In particular see PyPy's sandboxed VM and the Zope sandbox.

Devin Jeanpierre
  • 92,913
  • 4
  • 55
  • 79