Tensorflow allocating GPU memory when using tf.device('/cpu:0')

Question

System Info: 1.1.0, GPU, Windows, Python 3.5, code runs in ipython consoles.

I am trying to run two different Tensorflow sessions, one on the GPU (that does some batch work) and one on the CPU that I use for quick tests while the other works.

The problem is that when I spawn the second session specifying with tf.device('/cpu:0') the session tries to allocate GPU memory and crashes my other session.

My code:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import time

import tensorflow as tf

with tf.device('/cpu:0'):
  with tf.Session() as sess:
    # Here 6 GBs of GPU RAM are allocated.
    time.sleep(5)

How do I force Tensorflow to ignore the GPU?

UPDATE:

As suggested in a comment by @Nicolas, I took a look at this answer and ran

import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import tensorflow as tf

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

which prints:

[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 2215045474989189346
, name: "/gpu:0"
device_type: "GPU"
memory_limit: 6787871540
locality {
  bus_id: 1
}
incarnation: 13663872143510826785
physical_device_desc: "device: 0, name: GeForce GTX 1080, pci bus id: 0000:02:00.0"
]

It seems to me that even if I explicitly tell the script to ignore any CUDA devices, it still finds and uses them. Could this be a bug of TF 1.1?

Answer 1

It turns out that setting CUDA_VISIBLE_DEVICES to the empty string does not mask the CUDA devices visible to the script.

From the documentation of CUDA_VISIBLE_DEVICES (emphasis added by me):

Only the devices whose index is present in the sequence are visible to CUDA applications and they are enumerated in the order of the sequence. If one of the indices is invalid, only the devices whose index precedes the invalid index are visible to CUDA applications. For example, setting CUDA_VISIBLE_DEVICES to 2,1 causes device 0 to be invisible and device 2 to be enumerated before device 1. Setting CUDA_VISIBLE_DEVICES to 0,2,-1,1 causes devices 0 and 2 to be visible and device 1 to be invisible.

It seems like the empty string used to be handled as "no valid devices exist" but changed meaning, as it is not mentioned in the documentation.

Changing the code to os.environ["CUDA_VISIBLE_DEVICES"] = "-1" fixes the problem. Running

import os
os.environ["CUDA_VISIBLE_DEVICES"]="-1"    
import tensorflow as tf

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

now prints

[name: "/cpu:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 14097726166554667970
]

and instantiating a tf.Session does not hog GPU memory anymore.

Answer 2

Would you mind trying one of these config options ?

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
# or config.gpu_options.per_process_gpu_memory_fraction = 0.0
with tf.Session(config=config) as sess:
    ...

As per the documentation, it should help you manage your GPU memory for this particular session and so your second session should be able to run on GPU.

EDIT: according this answer you should also try this:

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"]="-1"