F1 score for padded outputs in Keras

Question

I have and LSTM sequence tagger in Keras which I use for highly unbalanced data. Therefore, I'd like to use the (multiclass) F1-score as the model's main metric. I have 2 questions:

1) I use zero-padding in the data (and thus mask_zero=True in my embeddings), and all the losses are computed for masked data automatically. However, I suppose that masking has to be done manually for custom metrics computation? Is there an efficient vectorized solution for that?

2) Is it possible to pass sklearn's f1_score implementation into the model's compile (maybe after wrapping it in some way)? Right off the bat, it didn't work because apparently a placeholder was passed into it rather than a numpy array (I use tensorflow backend..)

[UPD] Given my implementation, there's now this question: I'm not sure whether there's a possibility to have the output of the model masked as well. Because if we don't care about the model's output for the 'pad' input positions (they don't contribute to the loss anyway), there may as well be some random garbage in the output which will affect the F1 metric. It would be ideal to only have there zeros as well.

score 2 · Accepted Answer · answered Apr 14 '18 at 14:33

Switched to the following (based on this code):

import numpy as np
from keras.callbacks import Callback
from sklearn.metrics import f1_score


class ZeroPaddedF1Score(Callback):
    def on_train_begin(self, logs={}):
        self.val_f1s = []


    def on_epoch_end(self, epoch, logs={}):
        y_true = np.argmax(self.validation_data[1], axis=-1)
        y_pred = np.argmax(self.model.predict(self.validation_data[0]), axis=-1)
        val_f1 = zero_padded_f1(y_true, y_pred)
        self.val_f1s.append(val_f1)
        print ' - val_f1: %f' % (val_f1)


def zero_padded_f1(y_true, y_pred):
    y_pred_flat, y_true_flat = [], []
    for y_pred_i, y_true_i in zip(y_pred.flatten(), y_true.flatten()):
        if y_true_i != 0:
            y_pred_flat.append(y_pred_i)
            y_true_flat.append(y_true_i)
    result = f1_score(y_true_flat, y_pred_flat, average='macro')
    return result

It won't probably work with model.compile (because it operates with numpy arrays and thus an already compiled model), but it does the job as a callback.

what if the model had wrongly predicted pad as the class output for a time step before the actual padding had started — raviTeja, Mar 21 '21 at 06:19

score 0 · Answer 2 · answered Apr 14 '18 at 02:15

OK, here's my try. Reviews are very welcome!

Main F1 score logic is taken from here. For both y_pred and y_true coming as 3D tensors of the shape (batch_size, sequence_length, classes_number), we calculate single-class F1's over their corresponding slices, and then average the result. Class 0 is reserved for padding and does not contribute to the score.

from keras import backend as K

def precision(y_true, y_pred):
    """Precision metric.

    Only computes a batch-wise average of precision.

    Computes the precision, a metric for multi-label classification of
    how many selected items are relevant.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
    precision = true_positives / (predicted_positives + K.epsilon())
    return precision


def recall(y_true, y_pred):
    """Recall metric.

    Only computes a batch-wise average of recall.

    Computes the recall, a metric for multi-label classification of
    how many relevant items are selected.
    """
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
    recall = true_positives / (possible_positives + K.epsilon())
    return recall


def f1_binary(y_true, y_pred):
    p = precision(y_true, y_pred)
    r = recall(y_true, y_pred)
    return 2 * ((p * r) / (p + r + K.epsilon()))


def f1(classes_number, y_true, y_pred):
    result = 0.0
    for class_id in xrange(1, classes_number + 1):
        y_true_single_class = y_true[:,:,class_id]
        y_pred_single_class = y_pred[:,:,class_id]
        f1_single = f1_binary(y_true_single_class, y_pred_single_class)
        result += f1_single / float(classes_number)
    return result

And here's how to use it with a Keras model (classes_number argument is bound via the wrapped_partial):

model.compile(optimizer=opt,
              loss='categorical_crossentropy',
              metrics=[wrapped_partial(f1, classes_number)])

F1 score for padded outputs in Keras

2 Answers2