Inconsistent results in CNN using keras

Question

I have done a prediction for car damages whether they are severe or not based on images in Keras using CNN. Predicted class and accuracy changes every time I run the code for the same dataset and with no other parameters changed. I have tried restarting the kernal and also setting seed for the model with a hope of getting consistent results. I am new to python, so kindly help me in the getting same results every time.

import random
random.seed(801)
# Importing the Keras libraries and packages
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers import Dropout

# Initialising the CNN
classifier = Sequential()

# Step 1 - Convolution
classifier.add(Conv2D(64, (2, 2), input_shape = (64, 64, 3), activation = 'relu'))

# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size = (2, 2)))



# Adding a second convolutional layer
classifier.add(Conv2D(64, (2, 2), activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))





# Step 3 - Flattening
classifier.add(Flatten())


# Adding dropout
classifier.add(Dropout(0.2))

# Step 4 - Full connection
classifier.add(Dense(units = 128, activation = 'relu'))

# Adding dropout
classifier.add(Dropout(0.2))

classifier.add(Dense(units = 1, activation = 'sigmoid'))

# Compiling the CNN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

# Part 2 - Fitting the CNN to the images

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(rescale = 1./255,
                                  # shear_range = 0.2,
                                  # zoom_range = 0.2,
                                   horizontal_flip = True)

test_datagen = ImageDataGenerator(rescale = 1./255)

#train_labels = keras.utils.to_categorical(train_labels,num_classes)
#test_labels = keras.utils.to_categorical(test_labels,num_classes)

training_set = train_datagen.flow_from_directory('C:/Users/Allianz/Desktop/Image Processing/car-damage-detective-neokt/app/2 category/training',
                                                 target_size = (64, 64),
                                                 batch_size = 16,
                                                 class_mode = 'binary')


test_set = test_datagen.flow_from_directory('C:/Users/Allianz/Desktop/Image Processing/car-damage-detective-neokt/app/2 category/validation',
                                            target_size = (64, 64),
                                            batch_size = 16,
                                            class_mode = 'binary')

batch_size=16

classifier.fit_generator(training_set,
                         steps_per_epoch = 605//batch_size,
                         epochs = 9,
                         validation_data = test_set,
                         validation_steps = 5//batch_size
                         )

#classifier.save('first_model.h5')
classifier.save('first.h5')



# finding the number associated classes 
#classes=training_set.class_indices
#print(classes)

# extracting file names of images
import os
from PIL import Image
import numpy as np
path='C:/Users/Allianz/Desktop/Image Processing/car-damage-detective-neokt/app/data3a_full/validation/01-minor'
img_names = [f for f in os.listdir(path) if os.path.splitext(f)[-1] == '.JPEG']
#print(img_names[1])
img_names=np.asarray(img_names) #converting list to array


# predicting classes for multiple images
import numpy as np
from keras.models import load_model
from keras.preprocessing import image

#os.chdir('C:/Users/Allianz/Desktop/Image Processing/car-damage-detective-neokt/app/2nd check/pred')
os.chdir('C:/Users/Allianz/Desktop/Image Processing/car-damage-detective-neokt/app/data3a_full/validation/01-minor')
a=load_model('first.h5')
classes=[]
result=[]
for i in range(len(img_names)):

    img=image.load_img(img_names[i],
                   target_size=(64,64))
    test_image = image.img_to_array(img)
    test_image = np.expand_dims(test_image, axis = 0)
    result = a.predict(test_image)
    #print(result)
    if result[0] >= 0.5:
        prediction = 'severe'
    else:
        prediction = 'not severe'
    classes.append(prediction)
#print(classes) 

#prediction2=print(classes)

import pandas as pd

dfn=pd.DataFrame({'image':img_names,
                     'prediction':classes
                    })

len(dfn.loc[dfn['prediction']=='not severe'])
len(dfn.loc[dfn['prediction']=='severe'])

Is that one file? Or is the classification and training scripts separate? — ZWang, Feb 22 '19 at 13:56
Both training and classification are coded in a single script file. — Kamakshi, Feb 23 '19 at 10:40

score 1 · Answer 1 · answered Feb 24 '19 at 11:17

It looks like you're training the model every time you classify! This is what's causing the inconsistency. The reason why this yields different results, despite you setting the seed, can be found (here)[Why can't I get reproducible results in Keras even though I set the random seeds?.

I suggest you separate the two files so that you train in one script and load then test in another. This way you will get more consistent results.

score 0 · Answer 2 · answered Feb 22 '19 at 15:23

I had similar problems with loading weights. The problem is that when you load the weights keras radomly assigns the weights because of the model declaration. I switched to using checkpoints for storing my weights and model.load_weights(checkpoints_directory) to load the weights. You will have to use a callback for this. Here is a short code snippet for this task (Google has a nice video on his topic).

from keras.callbacks import ModelCheckpoint

callbacks = [ModelCheckpoint(checkpoints_directory, monitor='val_loss', save_weights_only=True, save_best_only=True, period=period)]

model.fit(..., callbacks=callbacks, ...)

Even after trying the above code snippet i am getting different prediction and accuracy everytime. Kindly guide me with any other alternative solution — Kamakshi, Feb 24 '19 at 15:50

Inconsistent results in CNN using keras

2 Answers2