0

I am running a large model on tensorflow using Keras and toward the end of the training the jupyter notebook kernel stops and in the command line I have the following error:

 2017-08-07 12:18:57.819952: E tensorflow/stream_executor/cuda/cuda_driver.cc:955] failed to alloc 34359738368 bytes on host: CUDA_ERROR_OUT_OF_MEMORY

This I guess is simple enough - I am running out of memory. I have 4 NVIDIA 1080ti GPUs. I know that TF uses only one unless specified. Therefore, I have 2 questions:

  1. Is there a good working example of how to utilise all GPUs in Keras

  2. In Keras, it seems it is possible to change gpu_options.allow_growth=True, but I cannot see exactly how to do this (I understand this is being a help-vampire, but I am completely new to DL on GPUs)

see CUDA_ERROR_OUT_OF_MEMORY in tensorflow

talonmies
  • 70,661
  • 34
  • 192
  • 269
GhostRider
  • 2,109
  • 7
  • 35
  • 53

1 Answers1

1

  1. See this Official Keras Blog

  2. Try this:

    import keras.backend as K

config = K.tf.ConfigProto()
config.gpu_options.allow_growth = True
session = K.tf.Session(config=config)
marcopah
  • 826
  • 2
  • 7
  • 25