2

I try to follow the tensorflow API 1.4 document to achieve what I need in a learning process.

I am now at this stage, can produce a predict object for example:

classifier = tf.estimator.DNNClassifier(feature_columns=feature_cols,hidden_units=[10, 20, 10], n_classes=3, model_dir="/tmp/xlz_model")

predict = classifier.predict(input_fn=input_pd_fn_prt (test_f),predict_keys=["class_ids"])
label =tf.constant(test_l.values, tf.int64)

how can I use predict and label in tf.metrics.auc for example:

out, opt = tf.metrics.auc(label, predict)

I have tried so many different options. there are no clear documentation how these tensorflow APIs can be should be used.

David Parks
  • 30,789
  • 47
  • 185
  • 328
Robert Zhou
  • 21
  • 3

1 Answers1

3

The function returns 2 operations:

auc, update_op = tf.metrics.auc(...)

If you run sess.run(auc) you will get back the current auc value. This is the value you want to report on, for example, print sess.run([auc, cost], feed_dict={...}).

The AUC metric may need to be computed over many calls to sess.run. For example, when the dataset you're computing the AUC for doesn't fit in memory. That's where the update_op comes in. You need to call it each time to accumulate the values needed to compute auc.

So during a test set evaluation, you might have this:

for i in range(num_batches):
    sess.run([accuracy, cost, update_op], feed_dict={...})

print("Final (accumulated) AUC value):", sess.run(auc))

When you want to reset the accumulated values (before you re-evaluate your test set, for example) you should re-initialize your local variables. The tf.metrics package wisely adds its accumulator variables to the local variables collection, which don't include trainable variables such as weights by default.

sess.run(tf.local_variables_initializer())  # Resets AUC accumulator variables

https://www.tensorflow.org/api_docs/python/tf/metrics/auc

David Parks
  • 30,789
  • 47
  • 185
  • 328
  • David, thank you for your comments. What I try to do right now is to find out how tensorflow API, architecture work. So testing the code with tf.InteractiveSession(). I run and debug: pr = classifier.predict(input_fn=input_pd_fn_prt (test_f)) I can see the values in pr. Then I try to use tf.metrics.auc with pr and label to get the auc information. then at the stage, I get error: AttributeError: 'generator' object has no attribute 'dtype'. accuracy, update_op = tf.metrics.auc(label, pr) The predict should return estimated results in pr, right? how I can use pr infor for tf.metrics.auc. – Robert Zhou Nov 11 '17 at 17:07
  • No, your predictions aren't overwritten by `tf.metrics.auc`. The `tf.metrics.auc` function creates new variables that accumulate `true_positives`, `true_negatives`, and `false_positives`. These variables are not being returned to you, but if you displayed your graph in tensorboard you would see them. When you request the `auc` tensor (a computed value, not a variable), it computes the auc metric from these accumulator variables. The `update_op` operation computes the tp, tn, fp, from `pr` and `label` and updates the accumulator variables appropriately. At no time is `pr` or `label` changed. – David Parks Nov 11 '17 at 19:26
  • Yes, thanks. I think tf.metrics.auc as an API, you pass label and predicted values from the model into, it returns auc value. My question is how to extract predicted values from the estimator models such as DNNClassifier, which can be used for tf.metrics.auc. In the tensorflow API doc, it doesn't document what the predict() function returns and how to use the returned values. For example, I need to debug to find out the DNNClassifier predict() returns probabilities, class_ids, classes ... Other people can only use DNNClassifier by finding some examples, or digging into the implementation. – Robert Zhou Nov 13 '17 at 18:13
  • Tensorflow is a general purpose library, it doesn't work like SciKit Learn, there's no high-level predict function, that would restrict you to a specific implementation. Your predicted values are an OP at the output layer of your network. If you use a softmax/cross entropy loss function you're probably passing in the top layer of the network to `tf.softmax_cross_entropy_with_logits`, to get the predictions just use the value you passed in there and apply `tf.nn.softmax` to it to get your predicted distribution. – David Parks Nov 13 '17 at 18:29
  • Then you usually run `tf.argmax` on that to get the specific predicted label. Basically, you have to do the operations yourself in tensorflow. It's not done for you anywhere. – David Parks Nov 13 '17 at 18:29
  • David, thank you for the explanation. I know what tensorflow is supposed to use for. We need to build our own models using it's APIs. I wish there is some kind of documents, documenting the underline architecture of tensorflow. – Robert Zhou Nov 14 '17 at 17:05