3

I'm trying to calculate F1 score in a tf.Estimator setup.

I've seen this SO question, but couldn't distill a working solution from it.

The thing with tf.Estimator is that it expects me to deliver a value and an update op, so right now, I have this piece of code at the end of my model:

if mode == tf.estimator.ModeKeys.EVAL:
    with tf.variable_scope('eval'):
        precision, precision_update_op = tf.metrics.precision(labels=labels,
                                            predictions=predictions['class'],
                                            name='precision')

        recall, recall_update_op = tf.metrics.recall(labels=labels,
                                      predictions=predictions['class'],
                                      name='recall')

        f1_score, f1_update_op = tf.metrics.mean((2 * precision * recall) / (precision + recall), name='f1_score')

        eval_metric_ops = {
            "precision": (precision, precision_update_op),
            "recall": (recall, recall_update_op),
            "f1_score": (f1_score, f1_update_op)}

Now the precision and recall seem to be working just fine, but on the F1 score, I keep getting nan.

How should I go about getting this to work?

EDIT:

A working solution can be achieved with tf.contrib.metrics.f1_score but since contrib is going to be deprecated in TF 2.0, I'd appreciate a contrib-less solution

bluesummers
  • 11,365
  • 8
  • 72
  • 108

4 Answers4

1

I did it in this way:

def f1_score_class0(labels, predictions):
    """
    To calculate f1-score for the 1st class.
    """
    prec, update_op1 = tf.compat.v1.metrics.precision_at_k(labels, predictions, 1, class_id=0)
    rec,  update_op2 = tf.compat.v1.metrics.recall_at_k(labels, predictions, 1, class_id=0)

    return {
            "f1_Score_for_class0":
                ( 2*(prec * rec) / (prec + rec) , tf.group(update_op1, update_op2) )
    }
mosen sayyadi
  • 11
  • 1
0

1) Why are you doing tf.metrics.mean? recall and precision are scalar values

2) have you tried printing f1_score and f1_update_op?

3) From the documentation of recall they mention how

For estimation of the metric over a stream of data, the function creates an update_op that updates these variables and returns the recall. update_op weights each prediction by the corresponding value in weights

Since you're getting the F1-score directly from the two ops that handle the updating, try doing a tf.identity (which effectively causes no change)

IanQ
  • 1,831
  • 5
  • 20
  • 29
  • You can't do that, because the `eval_metric_ops` expect to get a tuple of `(value, update_op)` – bluesummers Dec 04 '18 at 20:11
  • Yeah, so the first value is the f1_score as you've calculated, and the second op is just the identity op? – IanQ Dec 04 '18 at 20:43
0

The f1 value tensor can be computed from precision and recall values. The metrics must be (value, update_op) tuples. We can pass tf.identity for f1. This worked for me:

import tensorflow as tf

def metric_fn(labels, logits):
    predictions = tf.argmax(logits, axis=-1)
    pr, pr_op = tf.metrics.precision(labels, predictions)
    re, re_op = tf.metrics.recall(labels, predictions)
    f1 = (2 * pr * re) / (pr + re)
    return {
        'precision': (pr, pr_op),
        'recall': (re, re_op),
        'f1': (f1, tf.identity(f1))
    }
Bohumir Zamecnik
  • 2,450
  • 28
  • 23
  • What is the meaning of the `tf.identity(f1)` where the op is expected? – bluesummers Jul 15 '19 at 17:47
  • It converts a Tensor to a dummy Operation since the tf.metrics must return tuple (value Tensor, update Operation). – Bohumir Zamecnik Jul 16 '19 at 13:11
  • Are you sure this works? from my understanding the op is used for the ability to calculate the metric in batches, since here the op doesn't have any meaning, I wonder if it doesn't corrupt the score – bluesummers Jul 16 '19 at 13:30
0

TensorFlow addons already has an official solution

https://www.tensorflow.org/addons/api_docs/python/tfa/metrics/F1Score

bluesummers
  • 11,365
  • 8
  • 72
  • 108