10

I am running several tensorflow inferences using sess.run() in a loop and it happens that some inferences are too heavy for my GPU.

I get errors like :

2019-05-23 15:37:49.582272: E tensorflow/core/common_runtime/executor.cc:623] 
Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [306] and type float

I would like to be able to catch these specific OutOfMemory errors but not other errors (which may be due to a wrong input format or a corrupted graph.)

Obviously, a structure similar to :

try:
   sess.run(node_output, feed_dict={node_input : value_input})
except:
    do_outOfMemory_specific_stuff()

does not work since other kind of errors will lead to a call to the do_outOfMemory_specific_stuff function.

Any idea how to catch these OutOfMemory errors ?

Joseph Budin
  • 1,299
  • 1
  • 11
  • 28
  • 1
    It should be a [`tf.errors.ResourceExhaustedError`](https://www.tensorflow.org/api_docs/python/tf/errors/ResourceExhaustedError). This was suggested in the duplicate [How to catch tf.errors.ResourceExhaustedError in tensorflow?](https://stackoverflow.com/q/52392502/1782792), but there was no further answer, can you try whether catching that works? – jdehesa May 23 '19 at 13:49
  • @jdehesa Yes it works, thank you. I think you can write that as an answer – Joseph Budin May 23 '19 at 13:58

1 Answers1

16

You should be able to catch it via:

...
except tf.errors.ResourceExhaustedError as e:
    ...

according to this documentation.

Error - Syntactical Remorse
  • 7,468
  • 4
  • 24
  • 45
  • Could you show us how to proceed then when I catch the error? how can I flush the memory? – Minions Nov 30 '19 at 10:50
  • 1
    @Ghanem checkout https://stackoverflow.com/a/44102727/8150685 or the answer above it. Either you can set it to grow or you will need to do it in batch sizes. Either way I don't know if you will use this exception unless you plan to implement a better ask forgiveness than permission type solution where you try to do it in one go but if it still runs out of memory than you batch it. – Error - Syntactical Remorse Nov 30 '19 at 15:03