Tfhub's "efficientdet d0" model is 15MB but when I apply it to even the smallest image, tensorflow consumes 1-4GB of GPU memory - even having prevented tensorflow from allocating all the GPU memory. The same happens with the similarly small "efficientdet lite1" and "ssd mobilenet v2" models. All three networks advertise efficient, scalable performance suitable for mobile and embedded applications.
For example:
import tensorflow as tf
import tensorflow_hub as hub
for gpu in tf.config.list_physical_devices('GPU'):
tf.config.experimental.set_memory_growth(gpu, True)
# Optionally: tf.config.experimental.set_virtual_device_configuration(gpu, [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=256)])
detector = hub.load("https://tfhub.dev/tensorflow/efficientdet/d0/1")
# At this point, nvidia-smi shows 311MiB used
z = tf.zeros([1,32,32,3], dtype=tf.dtypes.uint8)
print(detector(z))
# At this point, nvidia-smi shows 4361MiB used
# or 1163MiB if memory_limit=256 enabled.
This came as a surprise to me - I could understand if a 15MB model was compressed and one needed 10x as much GPU memory - but needing 290x as much is a pretty big difference. 4GB here, 4GB there and pretty soon you're talking about serious GPU memory.
As you can see from the code above, I'm not doing any training that would fill memory with gradients and metadata, and I'm not processing large batches or huge images.
Is this normal, or am I doing something wrong here?
