1

I want to get UAR (unweighted accuracy) from confusion matrix to monitor UAR of validation data. However, it is difficult to deal with tensor.

https://www.davidtvs.com/keras-custom-metrics/

I did refer to this site and try to create my own metrics in Keras. I am making the metrics by using the first method to use both ModelCheckpoint and EarlyStopping supported by Keras.

model.compile(loss='categorical_crossentropy',optimizer=adam, metrics=['accuracy', uar_accuracy])

However, I don't know how to define the uar_accuracy function.

            def uar_accuracy(y_true, y_pred):

            # Calculate the label from one-hot encoding
            pred_class_label = K.argmax(y_pred, axis=-1)
            true_class_label = K.argmax(y_true, axis=-1)


            cf_mat = tf.confusion_matrix(true_class_label, pred_class_label )

            diag = tf.linalg.tensor_diag_part(cf_mat)
            uar = K.mean(diag)

            return uar

This result returns the average of the number of data right for each class. But I do not want the average of the number of correct data, but I want the average of the correct probabilities for each class.

How can I implement it?

I have implemented the following for the numpy type, not the Tensor type using sklearn.metrics and collections library

            def get_accuracy_and_cnf_matrix(label, predict):
            uar = 0
            accuracy = []
            cnf_matrix = confusion_matrix(label, predict)
            diag=np.diagonal(cnf_matrix)
            for index,i in enumerate(diag):
                uar+=i/collections.Counter(label)[index]

            # cnf_marix (Number of corrects -> Accuracy)    
            cnf_matrix = np.transpose(cnf_matrix)
            cnf_matrix = cnf_matrix*100 / cnf_matrix.astype(np.int).sum(axis=0)
            cnf_matrix = np.transpose(cnf_matrix).astype(float)
            cnf_matrix = np.around(cnf_matrix, decimals=2)   

            # WAR, UAR
            test_weighted_accuracy = np.sum(label==predict)/len(label)*100
            test_unweighted_accuracy = uar/len(cnf_matrix)*100    
            accuracy.append(test_weighted_accuracy)
            accuracy.append(test_unweighted_accuracy)

            return np.around(np.array(accuracy),decimals=2), cnf_matrix
Jeonghwa Yoo
  • 167
  • 1
  • 12

1 Answers1

2

You can use tf.reduce_sum to compute the sum of each row in your confusion matrix. This corresponds to the total number of data points for each class. Then you divide the diagonal elements with this row sum to compute the ratio of correctly predicted examples per class.

def non_nan_average(x):
    # Computes the average of all elements that are not NaN in a rank 1 tensor
    nan_mask = tf.debugging.is_nan(x)
    x = tf.boolean_mask(x, tf.logical_not(nan_mask))
    return K.mean(x)


def uar_accuracy(y_true, y_pred):
    # Calculate the label from one-hot encoding
    pred_class_label = K.argmax(y_pred, axis=-1)
    true_class_label = K.argmax(y_true, axis=-1)

    cf_mat = tf.confusion_matrix(true_class_label, pred_class_label )

    diag = tf.linalg.tensor_diag_part(cf_mat)    

    # Calculate the total number of data examples for each class
    total_per_class = tf.reduce_sum(cf_mat, axis=1)

    acc_per_class = diag / tf.maximum(1, total_per_class)  
    uar = non_nan_average(acc_per_class)

    return uar
Anna Krogager
  • 3,528
  • 16
  • 23
  • I could not think of this simple thing, I kept thinking! Thank you very much. Have a good day. – Jeonghwa Yoo Jan 21 '19 at 08:19
  • 1
    Happy to help :-) If this answer or any other one solved your issue, please mark it as accepted. Have a good day too – Anna Krogager Jan 21 '19 at 09:04
  • I have a problem. To prevent NaN from happening, I added the following code. Then the result of UAR is wrong. Do you know why? total_per_class = tf.reduce_sum(cf_mat, axis=1) total_per_class = tf.cast(total_per_class, dtype=tf.float32) total_per_class = total_per_class + tf.convert_to_tensor(K.epsilon()) – Jeonghwa Yoo Jan 21 '19 at 11:09
  • I suppose NaN happened because you had a zero row so that total_per_class is 0 for one of the classes? In that case, I guess you should leave out that class since you have no data for it. The reason why your code doesn't work is that you replace 0 by epsilon and when you divide by epsilon in acc_per_class you get a huge and incorrect number – Anna Krogager Jan 21 '19 at 11:19
  • I just realized that you divide 0 by epsilon so you don't get a huge number, you just get 0. But when you take the mean over all rows this will still give a wrong result – Anna Krogager Jan 21 '19 at 11:29
  • Thank you for the quick response! However, even if there is no row with a value of 0, the value of nan will be displayed. I don't know why... – Jeonghwa Yoo Jan 21 '19 at 11:42
  • Do you have any idea from https://stackoverflow.com/questions/46663013/what-is-y-true-and-y-pred-when-creating-a-custom-metric-in-keras ? My shape of final layer is (None,4) – Jeonghwa Yoo Jan 21 '19 at 11:52
  • Probably you have a batch of data where one of the classes doesn't occur, which results in a row with only zeros in your confusion matrix. Maybe you can try taking the average of all non-NaN entries in acc_per_class instead of the average of all of them. I edited my answer above with this change. – Anna Krogager Jan 21 '19 at 12:14
  • Thank you so soooo much. As you say, it seems to happen when a row becomes zero in a batch. There are still very few errors, but the results are much better. I will worry more about why this error occurs. Thank you! – Jeonghwa Yoo Jan 21 '19 at 13:01
  • I noticed an error in my non_nan_average function. I have edited my answer once again, now it should really work :-) – Anna Krogager Jan 21 '19 at 13:04
  • So thank you! You're so kind and nice :) Unfortunately... the error is more bigger ... it is too difficult to me.. haha.. – Jeonghwa Yoo Jan 21 '19 at 13:28
  • I think there is an error because of batch size. I think that the values are calculated strangely where the batch size does not fall apart. https://stackoverflow.com/questions/39925462/keras-custom-metric-gives-incorrect-tensor-shape.. Anyway, thank you! – Jeonghwa Yoo Jan 22 '19 at 08:17