3

My CNN code in keras is as follows:

from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers import Dropout

classifier = Sequential()
#1st Conv layer
classifier.add(Convolution2D(64, (9, 9), input_shape=(64, 64, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(4,4)))
#2nd Conv layer
classifier.add(Convolution2D(32, (3, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2,2)))

#Flattening
classifier.add(Flatten())

# Step 4 - Full connection
classifier.add(Dense(units = 128, activation = 'relu'))
classifier.add(Dropout(0.1))
classifier.add(Dense(units = 128, activation = 'relu'))
classifier.add(Dropout(0.2))
classifier.add(Dense(units = 128, activation = 'relu'))
classifier.add(Dense(units = 2, activation = 'softmax'))

classifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

#Fitting dataset

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip = True)

test_datagen = ImageDataGenerator(rescale = 1./255)

training_set = train_datagen.flow_from_directory('dataset/training_set',
                                                 target_size = (64, 64),
                                                 batch_size = 32,
                                                 class_mode = 'categorical')

test_set = test_datagen.flow_from_directory('dataset/test_set',
                                            target_size = (64, 64),
                                            batch_size = 32,
                                            class_mode = 'categorical')

classifier.fit_generator(
        training_set,
        steps_per_epoch=(1341+3875)/32,
        epochs=15,
        validation_data=test_set,
        validation_steps=(234+390)/32)

Wherever I see the use of roc_curve from sklearn.metrics, it takes parameters like x_train, y_train, x_test, y_test which I know can be pandas DataFrames but in my case it is not the case. How do I plot the ROC curve and get AUC score for model training for CNNs like here?

GFTW
  • 145
  • 1
  • 1
  • 14

3 Answers3

3

I got it working. All I had to do was match the datatype of preds obtained from preds = classifier.predict(test_set) with the true_labels I got from labels = test_set. Preds is basically a numpy.ndarray containing single element lists which have np.float32 values. Conversion of labels to that same format and shape got the roc_curve working.

Also, I had to add a third variable threshold in fpr, tpr, threshold = roc_curve(true_labels, preds) so no ValueError: too many values to unpack error popped up.

GFTW
  • 145
  • 1
  • 1
  • 14
1

Actually if look at the docs of sklearn.metrics.roc_curve (and almost for every sklearn metric) they don't take the inputs of your model (images) as arguments, it just takes the true labels and the predicted label. So after you make the inference on the test set, which in keras (here i just guessing) is something like

preds = classifier.predict(batch)

You call roc_curve as

fpr, tpr = roc_curve(true_labels,preds)

Probablly you have to change the type though, beacuse they're are tensor.

EDIT : I've checked the keras documentation on flow_from_directory and yields an iterator over (x,y) = (images,labels) so if you want to do some kind of post-training analysis you should get the labels using something like this:

labels = []
for _,y in test_set:
    labels.extend(list(y))

And if you only have two classes, change the class_mode to binary

umbreon29
  • 223
  • 2
  • 8
  • Suppose I have two class names, "NORMAL" and "PNEUMONIA" for a chest X-ray dataset. In the classifier.predict(batch), I take batch to be the test set. Then, while calling roc_curve, what should be the type of true_labels? Is it a list of strings where each element is either 'NORMAL' or 'PNEUMONIA'? Or is it 0s and 1s where 0 is assigned to 'NORMAL' by default and vice versa. And how do I know what sequence the model followed for predicting each image? Is it the normal sequence you see as the images are arranged in File Explorer? – GFTW May 01 '20 at 01:06
  • About you first question, yes you should assign a numeric label to normal and pneumonia. Do you have the labels for your test set? Sometimes people refer to the test as set without labels, In this case you should slipt your train set into a train and a validation set. I'm not sure about your second question, do you mean the model maybe change the order of your images? If it's that the answer is no, the predicitions will correspond to the images in the same order. – umbreon29 May 01 '20 at 02:36
  • For the test set, all I have is a folder named 'test_set' and in it are two more folders named 'PNEUMONIA' and 'NORMAL'. Could you perhaps edit your answer and add the corresponding code to label encode them so I can get true_labels accurately? I'm not sure how to use LabelEncoder to automate the process of assigning 0s and 1s to all images. Note that 'test_set' has 'NORMAL' as the first folder and 'PNEUMONIA' as the second. And you got my second question right. – GFTW May 01 '20 at 04:44
  • I tried your solution but it gets stuck on processing in Jupyter Notebook. So I tried investigating test_set and it turned out that every element is a list of image, label tuples (image and label are 2D and 1D list, respectively). I managed to get a list of labels of 0s and 1s and flattened them to a list of only 0s and 1s and named it 'true_labels'. I used this 'true_labels' and preds from preds=classifier.predict(test_set) as fpr, tpr = roc_curve(true_labels, preds) but it gave me this error `ValueError: too many values to unpack (expected 2)` – GFTW May 01 '20 at 14:00
1

To compute ROC AUC, you need scores, not the result of final classification/decision.

Since your model has a softmax with two neurons (classifier.add(Dense(units = 2, activation = 'softmax'))) as its last layer, this is a kind of multi-class classification where the number of classes is 2. But, the function roc_curve is restricted to use in binary classification problems. So, you cannot use it with softmax.

You can replace your 2 neurons and softmax with one neuron and sigmoid. Then , it is safe to use roc_curve in a binary classification problem.

There is another function named roc_auc_score which has a argument multi_class that converts a multiclass classification problem into multiple binary problems. E.g., auc_roc = roc_auc_score(labels, classifier.predict(...), multi_class='ovr'). However, this only returns AUC score and it cannot help you to plot the ROC curve.

ashkan
  • 476
  • 6
  • 14