1

I recently tried running tensor flow object-detection with faster_rcnn_nas model on k80 graphics card, which got usable 11 GB memory. But still it crashed and appears that it required more memory based on the console errors.

My training data set has 1000 images of size 400X500 (approx.) and test data has 200 images of same size.

I am wondering what would be the approx memory needed for running the aster_rcnn_nas model or in general is it possible to know the memory requirements for any other model ?

BhanuKiran
  • 2,631
  • 3
  • 20
  • 36

1 Answers1

0

Tensorflow doesn't have an easy way to compute memory requirements (yet), and it's a bit of a job to work it out by hand. You could probably just run it on the CPU and look at the process to get a ballpark number.

But the solution to your problem is straightforward. Trying to pass 1000 images of size 400x500 is insanity. That would probably exceed the capacity of the largest 24GB GPUs. You can probably only pass through 10's or 100's of images per batch. You need to split up your batch and process it in multiple iterations.

In fact during training you should be taking a random sample of images and training on this (this is the "stochastic" part of gradient descent). This is known as the "batch size". For the test set you might get all 200 images to go through (since you don't run backprop), but if not then you'll have to split up the test set too (this is quite common).

David Parks
  • 30,789
  • 47
  • 185
  • 328
  • Regarding the batch size. The default value is 1. [link to config file](https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/faster_rcnn_nas_coco.config) – BhanuKiran Apr 12 '18 at 05:33
  • Oh, hmm, do you have a GPU with fairly low memory? That doesn't seem excessive, but maybe a 2-3GB GPU won't fit the model. Anyway, you have 2 approaches to checking I think: 1) fire up the tensorflow debugger (you will wrap the `sess` object with the CLI debugger and run like normal, it's easy), 2) run TF on the CPU and watch the memory usage of the process, this involves simply setting an env variable `CUDA_VISIBLE_DEVICES=-1` – David Parks Apr 12 '18 at 15:20
  • What about testing the object detction model via webcam..is 2gb gpu sufficient? – Fasty Dec 11 '19 at 13:47
  • Try using the allow-growth memory option to tell TF to not allocate all memory to the GPU, then you should be able to see how much memory is used by your model: https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory – David Parks Dec 11 '19 at 19:35