I'm defining a custom F1 metric in keras for a multiclass classification problem (in particular n_classes = 4 so the output layer has 4 neurons and a softmax activation function). The idea is to keep track of the true positives, false negatives and false positives so as to gradually update the f1 score batch after batch. The code is the following:
def compute_confusion_matrix(true, pred, K):
result = tf.zeros((K, K), dtype=tf.int32)
for i in range(len(true)):
result = tf.tensor_scatter_nd_add(tensor = result, indices=tf.constant([[true[i], pred[i]]]),
updates=tf.constant([1]))
return result
def f1_function(y_true, y_pred):
k = 4
y_pred_lab = np.argmax(y_pred, axis=1)
conf_mat= compute_confusion_matrix(y_true, y_pred_lab, K = k)
tp = tf.linalg.tensor_diag_part(conf_mat)
fp = tf.reduce_sum(conf_mat, axis = 0) - tp
fn = tf.reduce_sum(conf_mat, axis = 1) - tp
support = tf.reduce_sum(conf_mat, axis = 1)
return tp, fp, fn, support
The f1_function returns the true positives, false positives, false negatives and the support of each class exploiting the confusion matrix computed through the compute_confusion_matrix function. Even though these functions work when called separately, problems arise when called during the model fit.
The custom metric is defined by subclassing keras.metrics.Metric as follows:
class F1Metric(keras.metrics.Metric):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.f1_fn = f1_function
self.tp_count = self.add_weight("tp_count", initializer="zeros", shape = (4,))
self.fp_count = self.add_weight("fp_count", initializer="zeros", shape = (4,))
self.fn_count = self.add_weight("fn_count", initializer="zeros", shape = (4,))
self.support_total = self.add_weight("support_total", initializer = "zeros", shape = (4,))
def update_state(self, y_true, y_pred, sample_weight=None):
tp, fp, fn, support = self.f1_fn(y_true, y_pred)
print(tp)
self.tp_count.assign_add(tf.cast(tp, dtype=tf.float32))
self.fp_count.assign_add(tf.cast(fp, dtype=tf.float32))
self.fn_count.assign_add(tf.cast(fn, dtype=tf.float32))
self.support_total.assign_add(tf.cast(support, dtype=tf.float32))
def result(self):
precisions = self.tp_count / (self.tp_count + self.fp_count)
recalls = self.tp_count / (self.tp_count + self.fn_count)
f1 = tf.constant(2, dtype=tf.float32) * (precisions*recalls) / (precisions + recalls)
weighted_f1 = (f1 * self.support_total) / tf.reduce_sum(tf.cast(self.support_total, dtype=tf.float32))
return weighted_f1
when i use this metric in model.fit I get this error: TypeError: Scalar tensor has no len().
Any explanation for this problem? Thanks.
EDIT:
The problem above was due to the type of y_true passed to the f1_function which was <class 'tensorflow.python.framework.ops.EagerTensor'>. So I transformed it into a 1D array by doing: y_true = np.ravel(y_true).
However, during the fit it gives me the following error close to the end of the first epoch: "Cannot assign to variable tp_count:0 due to variable shape (4,) and value shape () are incompatible."
The only thing i noticed is that the length of y_true and y_pred is no longer the same as the batch size (32) but less than that. That limit should be given by the size of the training set so it shouldn't be the cause of the problem. Any idea?
