Theano: Cannot allocate memory error for "shape.eval()" but not "np.asarray"

Question

When I try to print the shape of a shared variable using shape.eval(), I get a "Cannot allocate memory" error. But when I convert it to a numpy array (thereby moving it to CPU), I can not only get the shape but also the array itself. See below where it is stopped in pdb, data has been loaded into CPU/RAM and then I am trying to print the shape of p1:

(Pdb) p1
W
(Pdb) type(p1)
<class 'theano.sandbox.cuda.var.CudaNdarraySharedVariable'>
(Pdb) p1.shape.eval()
Problem occurred during compilation with the command line below:
/usr/bin/g++ -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -fPIC -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -o /home/ubuntu/.theano/compiledir_Linux-3.13--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpe2w2zz/990b9feb030f7691b8412ea91249349d.so /home/ubuntu/.theano/compiledir_Linux-3.13--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/tmpe2w2zz/mod.cpp -L/usr/lib -lpython2.7
ERROR (theano.gof.cmodule): [Errno 12] Cannot allocate memory
2015-08-20 12:09:19,032 theano.gof.cmodule cmodule.py:1083 ERROR [Errno 12] Cannot allocate memory
*** OSError: [Errno 12] Cannot allocate memory
(Pdb) a1 = np.asarray(p1.eval())
(Pdb) a1.shape
(21504, 2048)
(Pdb) a1
array([[ 0.03940072, -0.03306604, -0.00602638, ..., -0.00931167,
         0.05510775,  0.02733043],
       [snip]
       ...,
       [-0.06175347,  0.03134078, -0.06079643, ..., -0.03223133,
         0.00347659,  0.03415193]], dtype=float32)

borrow is set to True for p1, but this should not matter since I am using GPU, right?

There is enough free space in memory:

$ free -m
             total       used       free     shared    buffers     cached
Mem:         15039      14815        224          0          1       4112
-/+ buffers/cache:      10701       4338
Swap:            0          0          0

This error is a RAM (as opposed to GPU memory) error, right? Why am I able to print the shape after getting the variable to CPU (using np.asarray), but shape.eval() prints an error?

What's a good way to address this issue (besides, may be, getting more RAM)?

score 1 · Answer 1 · edited May 23 '17 at 11:51

Found the issue. See here.

eval() invokes subprocess() (for compiling) which in turn invokes fork().

This process had a very large amount of virtual memory (vsize) allocated to it. Also, there was no swap configured on this host. Even though the actual memory needed to execute this eval() would have been tiny, I believe the fork failed in allocating virtual memory and hence the error.

Adding swap space made the issue go away. The swap space never got used, but I believe was sufficient to satisfy the virtual memory allocation.

Theano: Cannot allocate memory error for "shape.eval()" but not "np.asarray"

1 Answers1