How can I solve 'ran out of gpu memory' in TensorFlow

Question

I ran the MNIST demo in TensorFlow with 2 conv layers and a full-conect layer, I got an message that 'ran out of memeory trying to allocate 2.59GiB' , but it shows that total memory is 4.69GiB, and free memory is 3.22GiB, how can it stop with 2.59GiB? And with larger network, how can I manage gpu memory? I concern only how to make best use of the gpu memory and wanna know how it happened, not how to pre-allocating memory

Answer 1

I was encountering out of memory errors when training a small CNN on a GTX 970. Through somewhat of a fluke, I discovered that telling TensorFlow to allocate memory on the GPU as needed (instead of up front) resolved all my issues. This can be accomplished using the following Python code:

    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    sess = tf.Session(config=config)

Previously, TensorFlow would pre-allocate ~90% of GPU memory. For some unknown reason, this would later result in out-of-memory errors even though the model could fit entirely in GPU memory. By using the above code, I no longer have OOM errors.

Note: If the model is too big to fit in GPU memory, this probably won't help!

Answer 2

It's not about that. first of all you can see how much memory it gets when it runs by monitoring your gpu. for example if you have a nvidia gpu u can check that with watch -n 1 nvidia-smi command. But in most cases if you didn't set the maximum fraction of gpu memory, it allocates almost the whole free memory. your problem is lack of enough memory for your gpu. cnn networks are totally heavy. When you are trying to feed your network DO NOT do it with your whole data. DO this feeding procedure in low batch sizes.

Answer 3

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

TensorFlow provides two Config options on the Session to control this.

The first is the allow_growth option, which attempts to allocate only as much GPU memory based on runtime allocations:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

The second method is the per_process_gpu_memory_fraction option, which determines the fraction of the overall amount of memory that each visible GPU should be allocated. For example, you can tell TensorFlow to only allocate 40% of the total memory of each GPU by:

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config)

Answer 4

Tensorflow 2

As we don't have sessions anymore the solution is not longer viable.

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process.
In some cases, it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this. One of them is using set_memory_growth tf.config.experimental.set_memory_growth

For a full understanding, I recommend this link: Limiting GPU memory growth

Answer 5

From TensorFlow guide

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    # Restrict TensorFlow to only allocate 1GB of memory on the first GPU
    try:
        tf.config.experimental.set_virtual_device_configuration(gpus[0],
       [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
        logical_gpus = tf.config.experimental.list_logical_devices('GPU')
        print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
    except RuntimeError as e:
        # Virtual devices must be set before GPUs have been initialized
        print(e)

Adjust memory_limit=*value* to something reasonable for your GPU. e.g. with 1070ti accessed from Nvidia docker container and remote screen sessions this was memory_limit=7168 for no further errors. Just need to make sure sessions on GPU cleared occasionally (e.g. Jupyter Kernel restarts).

Answer 6

For Tensorflow 2:

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)

Answer 7

TensorFlow Dataset objects. This is a high-performance option that is more suitable for datasets that do not fit in memory and that are streamed from disk or from a distributed filesystem.

If you have a large dataset and you are training on GPU(s), consider using Dataset objects, since they will take care of performance-critical details, such as:

Asynchronously preprocessing your data on CPU while your GPU is busy, and buffering it into a queue. Prefetching data on GPU memory so it's immediately available when the GPU has finished processing the previous batch, so you can reach full GPU utilization.

• tf.keras.preprocessing.image_dataset_from_directory turns image files sorted into class-specific folders into a labeled dataset of image tensors.

Resource : https://keras.io/getting_started/intro_to_keras_for_engineers/

Answer 8

for reducing the batch size to 8 from 32 works

Answer 9

Before dwelving into other possible explanations like the ones mentioned above, please check that there is no hung process reserving GPU memory. It has just happened to me that my Tensorflow script got hung on some error but I did not notice it because I monitored running processes with nvidia-smi. Now that hung script did not show up in nvidia-smi's output but was still reserving GPU memory. Killing the hung scripts (Tensorflow typically spawns as many as there are GPUs in the system) completely solved a similar problem (after having exhausted all the TF wizardry).

Answer 10

For Tensorflow 2 or Keras:

from tensorflow.python.framework.config import set_memory_growth
tf.compat.v1.disable_v2_behavior()
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        for gpu in gpus:
            set_memory_growth(gpu, True)
    except RuntimeError as e:
        print(e)

Answer 11

I realize you guys are all TensorFlow fans and unlikely to try other frameworks... we addressed the GPU memory issue from the ground up when designing our system. We allocate as much as possible on the GPU, but if a few layers don't fit, then they go on the CPU. In particular we needed many DNNs loaded at once for evaluation once deployed in a medical device. To achieve this we have specific Eval libraries that reuse GPU memory as much as possible. When doing eval you don't need all the previous calculations for many of the layers so that can be dynamically freed/or reused directly if already big enough. Currently we have 5 image segmentation DNNs loaded and running simultaneously, any one of them completely fills the GPU when training, but all fit in 4GB in eval mode.