When a Theano script tries to obtain more GPU memory than currently available, it immediately crashes:
Error allocating 26,214,400 bytes of device memory (out of memory).
Driver report 19,365,888 bytes free and 1,073,414,144 bytes total
Is there any way to configure Theano so that it doesn't directly crash when a GPU memory allocation fails? E.g., it could instead retry every X seconds, and give up after Y tries.
(One use case: I have several Theano scripts running on the same GPU and using 90% of its memory only 5% of the time)
