Validation generator accuracy drop near to chance when I change its properties-- keras ImageDataGenerator, model.evaluate

Question

I am reading images from a directory hierarchy (flow_from_directory using generators from the ImageDataGenerator class). The model is a fixed parameter mobilenetv2 + a trainable softmax layer. When I fit the model to training data, accuracy levels are comparable for training and validation. If I play with the validation parameters or reset the generator, accuracy for the validation generator drops significantly using model.evaluate or if I restart fitting the model with model.fit. The database is a 3D view database. Relevant code:

'''

batch_size=16

rescaled3D_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, zoom_range=0.2, 
                                                                 shear_range=0.2,  
                                                                 horizontal_flip=True)             

train_gen =rescaled3D_gen.flow_from_directory(data_directory + '/train/', seed=3,
                                              target_size = (pixels, pixels), shuffle=True,
                                              batch_size = batch_size, class_mode='binary')

val_gen =rescaled3D_gen.flow_from_directory(data_directory + '/test/', seed=3,
                                            target_size = (pixels, pixels), shuffle=True,
                                            batch_size = batch_size, class_mode='binary')
#MODEL
inputs = tf.keras.Input(shape=(None, None, 3), batch_size=batch_size)
x = tf.keras.layers.Lambda(lambda img: tf.image.resize(img, (pixels,pixels)))(inputs)
x = tf.keras.layers.Lambda(tf.keras.applications.mobilenet_v2.preprocess_input)(x)

mobilev2 = tf.keras.applications.mobilenet_v2.MobileNetV2(weights = 'imagenet', input_tensor = x,
                                                          input_shape=(pixels,pixels,3),
                                                          include_top=True, pooling = 'avg')
#add a dense layer for task-specific categorization.
full_model = tf.keras.Sequential([mobilev2, 
                                tf.keras.layers.Dense(train_gen.num_classes, activation='softmax')])

for idx, layers in enumerate(mobilev2.layers):
    layers.trainable = False

mobilev2.layers[-1].trainable=True

full_model.compile(optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.0001), 
             loss = 'sparse_categorical_crossentropy',
             metrics=['accuracy'])
#start fitting
val_gen.reset()
train_gen.reset()

full_model.fit(train_gen, 
               steps_per_epoch = samples_per_epoch, 
               epochs=30,
               validation_data=val_gen,
               validation_steps = int(np.floor(val_gen.samples/val_gen.batch_size)))

good_acc_score = full_model.evaluate(val_gen, steps=val_gen.n//val_gen.batch_size)

'''

reproduce strangeness by doing something like this:

'''

val_gen.batch_size=4
val_gen.reset()
val_gen.batch_size=batch_size

'''

Then validation accuracy is automatically lower (perhaps to chance) during fit or evaluation

'''

bad_acc_score = full_model.evaluate(val_gen, steps=val_gen.n//val_gen.batch_size)

#or

full_model.fit(train_gen, 
               steps_per_epoch = samples_per_epoch, 
               epochs=1,
               validation_data=val_gen,
               validation_steps = int(np.floor(val_gen.samples/val_gen.batch_size)))

'''

score 0 · Answer 1 · answered Oct 23 '20 at 14:51

0

I think the lead answer from this board solves the problem I observed Keras: Accuracy Drops While Finetuning Inception

answered Oct 23 '20 at 14:51

Mark Dranias

25
1
5

score 0 · Accepted Answer · answered Oct 23 '20 at 16:46

here area few things you might try. You can eliminate the Lamda layers by changing the train_gen as follows

rescaled3D_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, zoom_range=0.2,shear_range=0.2, horizontal_flip=True,
            preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input)

You do not need the Lamda resize layer since you specify the target size in flow from directory. In the val_gen you have shuffle=True. This will shuffle the validation image order for each epoch. Better to set it to False for consistancy. In the code for mobilenet you have include_top=True and pooling='avg' When include_top is True the pooling parameter is ignored. Setting include_top=True leaves the top layer of your model with a dense layer of 1000 nodes and a softmax activation function. I would set include_top=False. That way the mobilenet output is a global pooling layer that can directly feed your dense categorization layer. In the generators you set the class_mode='binary'. but in model.compile you set the loss as sparse_categorical_crossentropy. This will work but better to compile with loss=BinaryCrossentropy. It is preferable to go through the validation samples exactly one time per epoch for consistancy. To do that the the batch size should be selected such that validation samples/batch_size is an integer and use that integer as the number of validation steps. The code below will do that for you.

b_max=80 # set this to the maximum batch size you will allow based on memory capacity
length=val_gen.samples       
batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and length/n<=b_max],reverse=True)[0]  
val_steps=int(length/batch_size)

Changing validation batch size can change the results of validation loss and accuracy. Generally a larger batch size will result in less fluctuation of the loss but can lead to a higher probability of getting stuck in a local minimum. Try these changes and see if there is less variance in the results.

Validation generator accuracy drop near to chance when I change its properties-- keras ImageDataGenerator, model.evaluate

2 Answers2