5

I have tensorflow's gpu version installed, as soon as I create a session, it shows me this log:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: name: GeForce GTX TITAN Black major: 3 minor: 5 memoryClockRate (GHz) 0.98 pciBusID 0000:01:00.0 Total memory: 5.94GiB Free memory: 5.31GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN Black, pci bus id: 0000:01:00.0)

And when I check my GPU memory usage, around 90% of it gets consumed.

Tensorflow documentation does not say anything about this. Does it take control of the gpu ? Why does it consume most of the memory ?

deepdebugging
  • 115
  • 1
  • 9
  • I don't know exactly what tf.Session() does, but about memory allocation of the GPU, give a look at answers and link in https://stackoverflow.com/questions/44278021/memory-estimation-for-convolution-neural-network-in-tensorflow/44278911#44278911 – Pietro Tortella Jun 01 '17 at 12:02
  • @PietroTortella Thanks for that. I have a question for you, I arrived at this post by trying to execute tensorflow, PyTorch, caffe in a single script. When I tried to make a Session as a global variable in the same script it gave me an error `current context was not created by the StreamExecutor` but I don't see it when I create this Session inside a function. Do you know what is going wrong ? This is why i would like to know what is Session doing exactly. – deepdebugging Jun 01 '17 at 12:22

1 Answers1

2

TensorFlow sessions allocate ~all GPU memory on startup, so they can bypass the cuda allocator.

Do not run more than one cuda-using library in the same process or weird things (like this stream executor error) will happen.

Alexandre Passos
  • 5,186
  • 1
  • 14
  • 19
  • Thanks for your answer. So is this like a tensorflow specific issue ? Because I have been running other machine learning libraries within the same script and I do not see any problems there. – deepdebugging Jun 02 '17 at 10:15
  • You might not see errors but it's likely something weird will happen with the cuda context. – Alexandre Passos Jun 02 '17 at 15:43