4

I'm using custom Estimator in Tensorflow. tf.metrics.accuracy seems to be working fine but not tf.metrics.auc, which always shows 0.5 in both the returned EstimatorSpec for evaluation and in training on Tensorboard.

Below is my code snippet of the model function:

def _model_fn(features, labels, mode, params):
    ......
    accuracy = tf.metrics.accuracy(labels=labels,
                               predictions=predicted_classes,
                               name='acc_op')
    auc = tf.metrics.auc(labels=labels,
                               predictions=predicted_classes,
                               name='auc_op')
    metrics = {'accuracy': accuracy, 'auc': auc}
    tf.summary.scalar('accuracy', accuracy[1])
    tf.summary.scalar('auc', auc[1])

    if mode == tf.estimator.ModeKeys.EVAL:
        # accuracy shows the right value; auc always shows 0.5
        return tf.estimator.EstimatorSpec(mode, loss=loss, eval_metric_ops=metrics)

    # the accuracy, loss during training are written to Tensorboard correctly.
    # But auc always has value 0.5.
    summary_hook = tf.train.SummarySaverHook(
        params['skip_step'],
        output_dir=params['model_dir'],
        summary_op=tf.summary.merge_all())
    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op, training_hooks=[summary_hook])

Anyone knows what goes wrong with AUC calculation? I've read the following question. The difference here is since I'm sticking to the high-level Estimator API, it doesn't use session explicitly. Do I need to initialize the local variables and if so, how can I modify my code? Thanks a lot!

how to use tf.metrics.__ with estimator model predict output

Prabhakar
  • 1,138
  • 2
  • 14
  • 30
yiping
  • 96
  • 10
  • [From the docs](https://www.tensorflow.org/api_docs/python/tf/metrics/auc), "predictions should be distributed approximately uniformly in the range [0, 1]". Is this true for your data? – kww Feb 14 '18 at 23:36
  • @KathyWu, yes, I plotted the histogram of the prediction. The value is strictly in (0,1). I think the problem is likely due to I might not have called AUC properly or I missed out some operation. – yiping Feb 15 '18 at 02:20
  • maybe this answer could help you https://stackoverflow.com/questions/39435341/how-to-calculate-auc-with-tensorflow – Yu Zh Jun 06 '18 at 07:41

0 Answers0