10

Recently I implemented a VGG-16 network using both Tensorflow and PyTorch, data set is CIFAR-10. Each picture is 32 * 32 RGB.

I use a 64 batch size in beginning, while I found PyTorch using much less GPU memory than tensorflow. Then I did some experiments and got a figure, which is posted below.

Figure : GPU memory use compare

After some researching, I known the tensorflow using BFC algorithm to manage memory. So it's can explain why tensorflow's memory using decreasing or increasing by 2048, 1024, ... MB and sometimes the memory use not increasing when batch size is bigger.

But I am still confused, why the memory use is lower when batch size is 512 than batch size is 384, 448 etc. which has a smaller batch size. The same as when batch size is from 1024 to 1408, and batch size is 2048 to 2688.

Here is my source code:

PyTorch:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16-pytorch.py

Tensorflow:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16.py

edit: I have two Titan XP on my computer, OS: Linux Mint 18.2 64-bit.

I determine GPU memory usage with command nvidia-smi.

My code runs on GPU1, which is defined in my code:

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

And I am sure there only one application using GPU1.

GPU memory usage can be determined by the application list below. For example, like the posted screen shot below, process name is /usr/bin/python3 and its GPU memory usage is 1563 MiB.

Figure : terminal nvidia-smi

lp5
  • 126
  • 6
  • 3
    How did you determine memory usage? By default, TensorFlow allocates [all available GPU memory](https://www.tensorflow.org/tutorials/using_gpu#allowing_gpu_memory_growth). – aseipel Nov 27 '17 at 09:09
  • @aseipel Thanks for your suggestion! I have updated my question. – lp5 Nov 27 '17 at 11:33
  • 1
    TensorFlow allocates all available GPU memory at the start, so measuring with nvidia-smi will not give an accurate measurement. Instead you should use RunMetadata for measurement: https://stackoverflow.com/questions/36123740/is-there-a-way-of-determining-how-much-gpu-memory-is-in-use-by-tensorflow – suharshs May 01 '18 at 04:04
  • Did you got the answer? I have the similar issue with my code. The same amount of GPU memory is allocated when I adopt batch size 128 and 256, it grows from 64 to 128 and 256 to 512. – Yanghoon Jun 22 '20 at 02:49

1 Answers1

1

As noted in the comments, by default TensorFlow always takes up all memory on a GPU. I assume you have disabled that function for this test, but it does show that the algorithms do not generally attempt to minimize the memory that is reserved, even if it's not all utilized in the calculations.

To find the optimal configuration for your device and code, TensorFlow often runs (parts of) the first calculation multiple times. I suspect that this included settings for pre-loading data onto the GPU. This would mean that the numbers you see happen to be the optimal values for your device and configuration.

Since TensorFlow doesn't mind using more memory, 'optimal' here is measured by speed, not memory usage.

Edwin van der Helm
  • 223
  • 2
  • 5