How to find the ROC curve and AUC score of CNN model (keras)

Question

I am new to deep learning. I am trying to generate ROC curve for the following code. I am using keras. The class size is 10 and the image are RGB image of size 100* 100* 3.

I went through [This link][1]. My problem is also same but I could not find the true labels. I am new in this field so please help me. I also looked on [This for true label][2].

The code snippet of my program is:

target_size=(100,100,3)

train_generator = train_datagen.flow_from_directory('path',
    target_size=target_size[:-1],
    batch_size=16,
    class_mode='categorical',
    subset='training',
    seed=random_seed)

valid_generator = ...

test_generator = ...
n_classes = len(set(train_generator.classes))



input_layer = keras.layers.Input(shape=target_size)

conv2d_1 = keras.layers.Conv2D(filters=64, kernel_size=(3,3), strides=1, padding='same', 
activation='relu',
                       kernel_initializer='he_normal')(input_layer)

batchnorm_1 = keras.layers.BatchNormalization()(conv2d_1)
maxpool1=keras.layers.MaxPool2D(pool_size=(2,2))(batchnorm_1)


conv2d_2 = keras.layers.Conv2D(filters=32, kernel_size=(3,3), strides=1, padding='same', 
activation='relu',
                       kernel_initializer='he_normal')(maxpool1)
batchnorm_2 = keras.layers.BatchNormalization()(conv2d_2)

maxpool2=keras.layers.MaxPool2D(pool_size=(2,2))(batchnorm_2)


flatten = keras.layers.Flatten()(maxpool2)
dense_1 = keras.layers.Dense(256, activation='relu')(flatten)

dense_2 = keras.layers.Dense(n_classes, activation='softmax')(dense_1)



model = keras.models.Model(input_layer, dense_3)

model.compile(optimizer=keras.optimizers.Adam(0.001),
      loss='categorical_crossentropy',
      metrics=['acc'])
model.summary()

model.fit_generator(generator=train_generator, validation_data=valid_generator,
            epochs=200)
            
score = model.evaluate_generator(test_generator)

print(score)

Now please help me in getting AUC score and ROC Curve. [1]: How to find the ROC curve and AUC score of this CNN model (keras) [2]: Getting true labels for keras predictions

Does this answer your question? [How to compute Receiving Operating Characteristic (ROC) and AUC in keras?](https://stackoverflow.com/questions/41032551/how-to-compute-receiving-operating-characteristic-roc-and-auc-in-keras) — Calimo, Nov 30 '20 at 07:38
See my comments [here](https://stackoverflow.com/questions/61529293/how-to-find-the-roc-curve-and-auc-score-of-this-cnn-model-keras?rq=1) — ashkan, Oct 26 '21 at 10:06

score 1 · Accepted Answer · answered Dec 03 '20 at 07:33

Add this code. Hope it works.

import numpy as np
from sklearn import metrics

x, y = test_generator.next()
prediction = model.predict(x)

predict_label1 = np.argmax(prediction, axis=-1)
true_label1 = np.argmax(y, axis=-1)

y = np.array(true_label1)

scores = np.array(predict_label1)
fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=9)
roc_auc = metrics.auc(fpr, tpr)


plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange',
 lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic (ROC)')
plt.legend(loc="lower right")
plt.show()

score 0 · Answer 2 · edited Nov 30 '20 at 08:22

The score function does not provide roc and auc score by default we have to calculate separately. You can check following code snipet to calculate roc and auc score and plot there values.

from sklearn.metrics import roc_curve
y_pred_keras = model.predict(X_test).ravel()
fpr_keras, tpr_keras, thresholds_keras = roc_curve(y_test, y_pred_keras)

from sklearn.metrics import auc
auc_keras = auc(fpr_keras, tpr_keras)

import matplotlib.pyplot as plt
plt.figure(1)
plt.plot([0, 1], [0, 1], 'k--')
plt.plot(fpr_keras, tpr_keras, label='Keras (area = {:.3f})'.format(auc_keras))
plt.plot(fpr_rf, tpr_rf, label='RF (area = {:.3f})'.format(auc_rf))
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')
plt.show()

I think you have a spelling error. `fpr_rf`, `tpr_rf`, `auc_rf`. Should they be `fpr_keras `, `tpr_keras `, `auc_keras `? Because otherwise python says they are undefined. Am I correct or did you define them somewhere else? — Edison, Jul 11 '22 at 10:37

How to find the ROC curve and AUC score of CNN model (keras)

2 Answers2