30

I have a TensorFlow model, and one part of this model evaluates the accuracy. The accuracy is just another node in the tensorflow graph, that takes in logits and labels.

When I want to plot the training accuracy, this is simple: I have something like:

tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
writer = tf.train.SummaryWriter('/me/mydir/', graph=sess.graph)

Then, during my training loop, I have something like:

for n in xrange(1000):
  ...
  summary, ..., ... = sess.run([summary_op, ..., ...], feed_dict)
  writer.add_summary(summary, n)
  ...

Also inside that for loop, every say, 100 iterations, I want to evaluate the validation accuracy. I have a separate feed_dict for this, and I am able to evaluate the validation accuracy very nicely in python.

However, here is my problem: I want to make another summary for the validation accuracy, by using the accuracy node. I am not clear on how to do this though. Since I have the accuracy node it makes sense that I should be able to re-use it, but I am unsure how to do this exactly, such that I can also get the validation accuracy written out as a separate scalar_summary...

How might this be possible?

Spacey
  • 2,941
  • 10
  • 47
  • 63
  • For some other solutions, I had a similar question [here](http://stackoverflow.com/questions/37146614/tensorboard-plot-training-and-validation-losses-on-the-same-graph). – golmschenk Feb 14 '17 at 13:45
  • This may answer your questions (without setting two summary dirs): http://stackoverflow.com/questions/34471563/logging-training-and-validation-loss-in-tensorboard – Wei Liu Mar 01 '17 at 21:25

2 Answers2

41

You can reuse the the accuracy node but you need to use two different SummaryWriters, one for the training runs and one for the test data. Also you have to assign the scalar summary for accuracy to a variable.

accuracy_summary = tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
summaries_dir = '/me/mydir/'
train_writer = tf.train.SummaryWriter(summaries_dir + '/train', sess.graph)
test_writer = tf.train.SummaryWriter(summaries_dir + '/test')

Then in your training loop you have the normal training and record your summaries with the train_writer. In addition you run the graph on the test set each 100th iteration and record only the accuracy summary with the test_writer.

# Record train set summaries, and train
summary, _ = sess.run([summary_op, train_step], feed_dict=...)
train_writer.add_summary(summary, n)
if n % 100 == 0:  # Record summaries and test-set accuracy
  summary, acc = sess.run([accuracy_summary, accuracy], feed_dict=...)
  test_writer.add_summary(summary, n)
  print('Accuracy at step %s: %s' % (n, acc))

You can then point TensorBoard to the parent directory (summaries_dir) and it will load both data sets.

This can be also found in the TensorFlow HowTo's https://www.tensorflow.org/versions/r0.11/how_tos/summaries_and_tensorboard/index.html

wih
  • 480
  • 5
  • 7
  • Thanks wih let me try this out...will report back. – Spacey Oct 20 '16 at 17:07
  • Ok so I think this is the issue: The "summary_op" contains the merged summaries of some things. However I just want the "summary" for the validation_error... does that make sense? Like if I pass in "summary_op" again inside the n % 100 loop and get the "summary" out, then it seems like it would "run" ALL the summaries. (All the ones that were merged). But I just want the accuracy node to be run... – Spacey Oct 20 '16 at 21:48
  • Also, why doesnt the test_writer have a "sess.graph" in it? – Spacey Oct 20 '16 at 21:55
  • The summary_op will write all summaries to the test file. I adjusted the code, now only the accuracy summary is recorded for test. – wih Oct 20 '16 at 23:15
  • I figured it out, but thanks for validating! This also answers a related question I had on overlaying the plots, which it now does automatically. Thanks so much!! :) – Spacey Oct 20 '16 at 23:18
  • The test_writer doesn't have the graph because it# – wih Oct 20 '16 at 23:18
  • 1
    Never press enter while commenting ;) The test_writer doesn't have the graph because it's the same graph and then tensorboard would display it twice. Displaying different "runs" (here it's the same run but different data) is the only way to overlay plots. When you modify your network it would be useful to display both graphs. – wih Oct 20 '16 at 23:24
  • I have a question regarding this which I am trying to solve. Suppose validation data set is big suppose has more than 2000 samples. Now, I can not feed the dictionary of all the samples. What I want to do is pass the validation set in batches of size 50, collect all the accuracy in an array, average those accuracies and then plot that on tensorboard. – Nishank Singla Apr 04 '18 at 20:59
  • It shows two equal graphs, each have two plot. How to remove one of these repetitions? – keramat Jul 02 '18 at 13:12
  • Could it be possible to merge only 2 operations(e.g. The test accuracy and test cost) into one test summary node? I'm looking for some statement similar to tf.merge_all_summaries() but with specific summary nodes. – StayFoolish Aug 15 '18 at 10:00
7

To run the same operation but get summaries with different feed_dict data, simply attach two summary ops to that op. Say you want to run accuracy op on both validation and test data and want to get summaries for both:

validation_acc_summary = tf.summary.scalar('validation_accuracy', accuracy)  # intended to run on validation set
test_acc_summary = tf.summary.scalar('test_accuracy', accuracy)  # intended to run on test set
with tf.Session() as sess:
    # do your thing
    # ...
    # accuracy op just needs labels y_ and input x to compute logits 
    validation_summary_str = sess.run(validation_acc_summary, feed_dict=feed_dict={x: mnist.validation.images,y_: mnist.validation.labels})
    test_summary_str = sess.run(test_acc_summary, feed_dict={x: mnist.test.images,y_: mnist.test.labels})

    # assuming you have a tf.summary.FileWriter setup
    file_writer.add_summary(validation_summary_str)
    file_writer.add_summary(test_summary_str)

Also remember you can always pull raw (scalar) data out of the protobuff summary_str like this and do your own logging.

Yibo Yang
  • 2,353
  • 4
  • 27
  • 40