Trying to add auc-roc score into CNN training

Question

My current CNN has relative high accuracy but low auc score, so I want to train my model considering both accuracy and auc. However, when I tried to add 'auc' as the second metrics to train, I cannot start my epochs.

This is the error message I am getting:

FailedPreconditionError: Error while reading resource variable conv2d_4/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv2d_4/kernel/N10tensorflow3VarE does not exist. [[{{node conv2d_4/Conv2D/ReadVariableOp}}]]

I have tried the function auc provided in previous discussions. Sorry I can't find the post now.

from keras import backend as K

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

auc_model = models.Sequential()
auc_model.add(layers.Conv1D (kernel_size = (200), filters = 10, input_shape=(1644,1) , activation='relu'))
auc_model.add(layers.MaxPooling1D(pool_size = (50), strides=(10)))
auc_model.add(layers.Reshape((40, 35, 1)))

auc_model.add(layers.Conv2D(16, (3, 3), activation='relu'))
auc_model.add(layers.Conv2D(16, (3, 3), activation='relu'))
auc_model.add(layers.MaxPooling2D((2, 2)))

auc_model.add(layers.Flatten())
auc_model.add(layers.Dense(32, activation='relu', kernel_regularizer=keras.regularizers.l2(0.001)))
auc_model.add(layers.Dropout(rate=0.2))
auc_model.add(layers.Dense(1, activation='sigmoid'))

auc_model.compile(optimizer='adam',
                       loss='binary_crossentropy',
                       metrics=['accuracy', auc])

auc_model.summary()


from tensorflow.keras.callbacks import EarlyStopping

target = y_tr.columns[0]
rows_tr = np.isfinite(y_tr[target]).values
rows_te = np.isfinite(y_te[target]).values

x_train = x_tr[rows_tr].reshape((x_tr[rows_tr].shape[0], 1644, 1))
x_test = x_te[rows_te].reshape((x_te[rows_te].shape[0], 1644, 1))

auc_model.fit( x_train, y_tr[target][rows_tr], 
              validation_data=(x_test, y_te[target][rows_te]), epochs = 5)

print('\n# Evaluate on test data')
results = auc_model.evaluate(x_test, y_te[target][rows_te], batch_size = 8, verbose=1)

I want to start my training process considering both accuracy and auc score. Thanks.

score 0 · Answer 1 · answered Jul 18 '19 at 07:09

Metric is just for reporting an evaluation of your trained model at each epoch. It does not change anything on your training.

If you want to make your model consider the AUC as well, you should modify your loss. Minimizing the loss of binary_crossentropy, naturally, maximize the accuracy without regarding the AUC. This makes it more problematic when you have an imbalanced data set, like one skewed class.

If you want it really only for the metric, you can see this post: How to compute Receiving Operating Characteristic (ROC) and AUC in keras?

But if you truly want your model to maximize the AUC, you should write a custom loss function on Keras and put it in the loss of your model. There is a good discussion here: https://www.kaggle.com/c/invasive-species-monitoring/discussion/32762

I have seen this discussion, so that is why I have the auc function. But the discussion is not that comprehensive, since I found it adds the auc function as a metric. I did the same but with the error message above. I cannot use conv1d, conv2d and dense. — PUMENG LYU, Jul 18 '19 at 14:52
Is y_tr a tensor? If it is, you do need this command before start fitting I believe: *init = tf.global_variables_initializer()* — alift, Jul 18 '19 at 18:06

Trying to add auc-roc score into CNN training

1 Answers1