1

I am using Bayesian regression for hyperparameter optimization, therefore, for some parameters I am running out of memory and this is throwing an error. Therefore, this error could be generated after many iterations, but might not be generated as well. So, I am looking to catch this error and then continue running the code normally. So, I am not sure where to place try and catch block in tensorflow. I guess, the last method throwing the error is session.py. Here is the error:

---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1321     try:
-> 1322       return fn(*args)
   1323     except errors.OpError as e:

...

ResourceExhaustedError: OOM when allocating tensor with shape[14,128,1,788200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Layer19/temporal_layer/bn_relu_conv_1/reduced_conv_19/conv1d/ExpandDims, PermConstNHWCToNCHW-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Any help is much appreciated!!

jdehesa
  • 58,456
  • 7
  • 77
  • 121
I. A
  • 2,252
  • 26
  • 65
  • Did you try to do a try/catch over a large part of the code, and the njust catch ResourceExhaustedError? – zwep Dec 11 '18 at 11:19
  • 1
    Possible duplicate of [How to raise an exception for a tensorflow out of memory error](https://stackoverflow.com/questions/56276627/how-to-raise-an-exception-for-a-tensorflow-out-of-memory-error) – jdehesa May 23 '19 at 15:14

0 Answers0