I have a python programme which sometimes consumes a lot of RAM.
I tried to restrict the usage by
resource.setrlimit(resource.RLIMIT_AS, (2**32, 2**32))
However, this causes Python to crash instead of raising a MemoryError.
How do I keep Python alive? I.e. I want the MemoryError.
The problem appears, when I try to solve a large instance in cvxpy. (This also means I cannot reduce memory usage of the code - it is not my code.) But when I allocate large amounts of RAM via
- numpy (
numpy.arange(1000*1000*250, dtype="u8"),
from How do I use `setrlimit` to limit memory usage? RLIMIT_AS kills too soon; RLIMIT_DATA, RLIMIT_RSS, RLIMIT_STACK kill not at all), or - shell (call
yes | tr \\n x | head -c $((1024*1024*1024*10)) | grep n,
from https://unix.stackexchange.com/questions/99334/how-to-fill-90-of-the-free-memory)
then i get the MemoryError as expected.
What is the difference?
Edit: The error message is:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Edit: I now have a (M)WE.
import cvxpy as cvx
import numpy as np
import random
import resource
resource.setrlimit(resource.RLIMIT_AS, (2**32, 2**32))
# Problem data.
m = 500
n = 500
np.random.seed(1)
random.seed(1)
A = np.random.randn(m, n)
b = np.random.randn(m, 1)
# Construct the problem.
X = cvx.Semidef(m)
objective = cvx.Minimize(cvx.sum_entries(cvx.norm(X)))
constraints = [sum([X[random.randint(0,499),random.randint(0,499)] for _ in range(50)]) >= random.random() for _ in range(5000)]
prob = cvx.Problem(objective, constraints)
print("Optimal value", prob.solve(solver = 'SCS'))
After a few seconds it starts taking up large amounts of memory (more than 1GB, which is supposed to be the limit) and then crashes with the above error.
