4

I am able to train on a local machine that has 4x 1080Ti's and as others have noted TF grabs all the available memory on my machine. After poking around for a bit most searches lead me to solutions for the base TF and not the Object Detection API, for example:

How to prevent tensorflow from allocating the totality of a GPU memory?

How do I access these sorts of options within the Object Detection API? How can I have similar TF style control over training within the OD API? Is there a proper way to within the OD API / slim API?

I tried adding a GPUOptions message to that training.proto but that didn't seem to make a difference.

Steve Goley
  • 193
  • 3
  • 9
  • I've found a workaround for now by defining CUDA_VISIBLE_DEVICES in the before I make the call to the object_detection/train.py script. This is a bit of a hack and its disappointing to lose a lot of the flexibility of TF when using the OD API, but it'll work for now. – Steve Goley Oct 09 '17 at 15:58

2 Answers2

5

I would like also to add the second choice for instructing tensorflow to use a portion of the available memory. According to this guide there are 2 options:

  • gpu_options.per_process_gpu_memory_fraction = 0.xxx

  • gpu_options.allow_growth

With the first one (as it has previously mentioned in the answers) a percentage of the whole available GPU memory is defined to be used while with the second you instruct tensorflow to use only as much memory as it's necessary to.

According to the documentation the first one should be somehow better if we know beforehand the necessary memory requirement since it allows for

... more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

On the other hand if we don't know the GPU memory requirements beforehand I would prefer the second one even with a slight performance sacrifice.

As for the actual use in tensorflow's recent API changes I have inserted them like this:

session_config = tf.ConfigProto()
session_config.gpu_options.allow_growth = True
# or use
# session_config.gpu_options.per_process_gpu_memory_fraction = 0.xxx

tf.logging.set_verbosity(tf.logging.INFO)
config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, session_config=session_config)

in main() in model_main.py.

Tested on tensorflow 1.12.0.

Eypros
  • 5,370
  • 6
  • 42
  • 75
2

I ran into a similar problem and added this attribute to the session configuration in trainer.py which reduced the amount of video memory used:

session_config.gpu_options.per_process_gpu_memory_fraction = 0.6

Effect confirmed using 

watch nvidia-smi
Eypros
  • 5,370
  • 6
  • 42
  • 75
Raf
  • 186
  • 1
  • 6
  • Should add the call to the session creation, also there's a type in the nvidia-smi call. You can also choose which GPU to use now in the gpu_options configuration. – Steve Goley Jul 20 '18 at 14:45