4

I have made a tensor flow model. But always get a NAN loss for some reason. I would like to know how I could debug and view each value in each tensor.

For eg:-

 out = tf.add(tf.matmul(outputs[-1], _weights['out']), _biases['out'])

During Run time I'd like to View the values in this tensor and see where things go wrong. I have found something similar in this post

Where they do something like this 

out = tf.add(tf.matmul(outputs[-1], _weights['out']), _biases['out'])
out = tf.Print(out, [out], message="This is softmax Output: ")

But this gives and out put like this 

I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [2.148583e-08 5.9002307e-08 -9.90654e-08...]
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]
Iter 64, Minibatch Loss= nan, Training Accuracy= 0.01562
Testing Accuracy: 0.0
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]
I tensorflow/core/kernels/logging_ops.cc:79] This is softmax Output: [nan nan nan...]

Which is not really that Useful as I cant view all the values. I'd like to know if there is a step by step debugging option ?

Community
  • 1
  • 1
Arsenal Fanatic
  • 3,663
  • 6
  • 38
  • 53
  • You could use similar recipe to print intermediate tensors as well – Yaroslav Bulatov Apr 11 '16 at 19:48
  • True but the problem is that this doesn't print the whole thing it prints the first few values in a tensor. And if I would like to perform some math tests on it like apply clipping and pass it to next tensor I couldn't do it. But yeah will use it as a last resort. – Arsenal Fanatic Apr 11 '16 at 19:53
  • There isn't a convenient way to do such step-by-step calculations atm although we are working on it. One work-around is to redefine everything as variables and use "tf.assign" to put values in them. This way you can examine their state between session runs – Yaroslav Bulatov Apr 11 '16 at 20:16

2 Answers2

1

TensorFlow now has a builtin debugger called tfdbg. It exposes the intermediate tensors in the graph, along with the graph structure, which should make it easier for you to debug this type of problems. Compared to print ops, it requires less code change and provides better coverage of the graph.

Please take a look at its documentation / tutorial at master HEAD: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/how_tos/debugger/index.md

Shanqing Cai
  • 3,756
  • 3
  • 23
  • 36
0

tf.Print() will display more results if you pass the summarize optional parameter. For example summarize=100 would display the first 100 elements of the tensor.

API Reference

(Not sure if the feature was there when the question was asked)

Mattias Arro
  • 1,102
  • 11
  • 16