CNN training system memory issue

Question

i am trying to train a CNN on CPU with a input of (102480,40,40,2) dimensions and output of (40,40). Before i trained a model using (102480,40,40) input and it worked perfectly. Now when i run the script systems memory usage increases until it needs more and my pc freezes. I only have 8 GB ram.

i tried adding this line to my code which did not help at all.

sess = tf.Session(config=tf.ConfigProto(inter_op_parallelism_threads=1, intra_op_parallelism_threads=1))

This is my model:

model = Sequential()
model.add(Conv2D(64,(3,3), input_shape = train_data.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Conv2D(64,(3,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))


model.add(Dense(1600))
model.add(Reshape((40,40)))
model.add(Activation('sigmoid'))

model.compile(loss='MSE',
             optimizer='SGD',
             metrics=['MAE'])

history = model.fit(train_data, train_target,
          batch_size=256,
          epochs=100,
          validation_split=0.1,
          callbacks = [cp_callback])

Can i somehow decrease the system memory usage without losing accuracy?

You are using way too high batch size. Decrease the batch size to something much much smaller like 8, or 16 and try again. — gorjan, Aug 13 '19 at 20:59

score 1 · Accepted Answer · answered Aug 25 '19 at 13:44

1

Instead of decreasing the batch size (which can impact the accuracy) I simply changed the data types from float64 to float32. This cut the memory usage by half.

answered Aug 25 '19 at 13:44

Olca Orakcı

372
3
12

Works like a charm! Store it in Float16 if you really need to squeeze it – ricardo Jul 07 '21 at 21:30

score 0 · Answer 2 · answered Aug 13 '19 at 21:59

The calculation of the model memory depends on the batch size. In fact, you can roughly estimate the memory usage by multiplying the batch size by the input shape. It's like you have 256 times the input size in your case. You can try to lower the batch size, but that may impact your accuracy. A way to minimize it's as stated in this answer: https://datascience.stackexchange.com/a/12533

In the case that you do need bigger batch sizes but it will not fit on your GPU, you can feed a small batch, save the gradient estimates and feed one or more batches, and then do a weight update. This way you get a more stable gradient because you increased your virtual batch size.

Finally, you can calculate the approximate memory usage using the solution of this answer https://stackoverflow.com/a/46216013/6598433 by ZFTurbo that I put below:

def get_model_memory_usage(batch_size, model):
    import numpy as np
    from keras import backend as K

    shapes_mem_count = 0
    for l in model.layers:
        single_layer_mem = 1
        for s in l.output_shape:
            if s is None:
                continue
            single_layer_mem *= s
        shapes_mem_count += single_layer_mem

    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

    number_size = 4.0
    if K.floatx() == 'float16':
         number_size = 2.0
    if K.floatx() == 'float64':
         number_size = 8.0

    total_memory = number_size*(batch_size*shapes_mem_count + trainable_count + non_trainable_count)
    gbytes = np.round(total_memory / (1024.0 ** 3), 3)
    return gbytes

CNN training system memory issue

2 Answers2