Memory problem when adding dense layer to CNN

Question

I am implementing my first CNN in Tensorflow and I am having trouble when adding the dense layer to my CNN model. Here is the code:

batch_size = 4
sample_shape = (batch_size, 24, 30, 30, 5)

model = models.Sequential()
model.add(layers.Conv3D(96, kernel_size=(4, 4, 4), activation='relu', padding='same', input_shape=sample_shape))
model.add(layers.Conv3D(64, kernel_size=(3, 3, 3), activation='relu', padding='same'))
model.add(layers.Conv3D(64, kernel_size=(1, 1, 5), activation='relu', padding='same'))
model.add(layers.Flatten())
model.add(layers.Dense(256, activation='relu'))

model.summary()

I am getting the following output. Later, my program crashes. What needs so much memory? It seems to be the Dense Layer, but I can't explain it.

2021-10-20 19:03:53.219849: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 5662310400 exceeds 10% of free system memory.

Your convolution layers probably need so much memory. Take a look at the number of their parameters. — AloneTogether, Oct 20 '21 at 17:21
The message you showed does look like a warning, not like an error. Are you sure that this is the reason why your program crashes? — Marc Felix, Oct 20 '21 at 17:22
Try `model.summary()` to see the number of parameters your model have, just by looking you have large number of 3D filters, check also [this question](https://stackoverflow.com/questions/50304156/tensorflow-allocation-memory-allocation-of-38535168-exceeds-10-of-system-memor). — Mohamed abdelmagid, Oct 20 '21 at 17:29

score 0 · Answer 1 · answered Oct 20 '21 at 17:39

Your flattened weight shape at the last layer is roughly

4 * (24 * 30 * 30 * 64 * 256) = 1,415,577,600

That's an insane amount of parameters. Use MaxPooling3D between convolutional layers or GlobalAveragePooling3D instead of Flatten to reduce the number of parameters.

Memory problem when adding dense layer to CNN

1 Answers1