No, this is impossible. Python doesn't keep track of which functions are responsible for allocating new memory, and nor does libc, so there's no way to limit memory usage just by a single function.
In order to do this you would have to modify Python so that you use a new function for allocating memory that requires it to be specified what Python function is responsible, so that it can reject the memory allocation if that function goes over its limit.
The only other way to do this would indeed be to execute the function in a separate process with limited memory, as @JBernardo said.
As for implementing sandboxes, there already are relatively well tested implementations. Is there any reason you can't use those? In particular see PyPy's sandboxed VM and the Zope sandbox.