1

I am trying to add the weights and biases to tensorboard according to the layers. The following way I tried: 

tf.reset_default_graph()

X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])
y = tf.placeholder(tf.float32, [None,n_outputs])


layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons, 
                                 activation=tf.nn.leaky_relu, use_peepholes = True)
         for layer in range(n_layers)]
# for i, layer in enumerate(layers):
#     tf.summary.histogram('layer{0}'.format(i), tf.convert_to_tensor(layer))


multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
for index,one_lstm_cell in enumerate(multi_layer_cell):
    one_kernel, one_bias = one_lstm_cell.variables
    # I think TensorBoard handles summaries with the same name fine.
    tf.summary.histogram("Kernel", one_kernel)
    tf.summary.histogram("Bias", one_bias)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)

stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
outputs = outputs[:,n_steps-1,:] # keep only last output of sequence

But I got the following error: 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-43-761df6e116a7> in <module>()
     44 
     45 multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
---> 46 for index,one_lstm_cell in enumerate(multi_layer_cell):
     47     one_kernel, one_bias = one_lstm_cell.variables
     48     # I think TensorBoard handles summaries with the same name fine.

TypeError: 'MultiRNNCell' object is not iterable

I would like to know what I have missed so that I can add the variables for visualization in the tensorboard. Kindly, help me.

Jaffer Wilson
  • 7,029
  • 10
  • 62
  • 139

1 Answers1

2

MultiRNNCell is indeed not iterable. For your case, first, the RNN variables will not be created until you call tf.nn.dynamic_rnn, so you should try to retrieve them after that. Second, with use_peephole you have more than kernel and bias variables. To retrieve them, you can access all the RNN variables together from multi_layer_cell.variables or each layer's own set through the cell objects stored in layers:

multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
for index, one_lstm_cell in enumerate(layers):
    one_kernel, one_bias, *one_peepholes = one_lstm_cell.variables
    tf.summary.histogram("Kernel", one_kernel)
    tf.summary.histogram("Bias", one_bias)
jdehesa
  • 58,456
  • 7
  • 77
  • 121
  • How I can have the weights displayed on the Tensorboard and whether the output is layer wise or is there something else? Kindly, enlighten me sir. – Jaffer Wilson Sep 05 '18 at 10:59
  • @JafferWilson Well that's another matter, it depends on what and how exactly you want to display. [`tf.summary.histogram`](https://www.tensorflow.org/api_docs/python/tf/summary/histogram) will make a histogram out of all the values in the tensor and after a few plots you would get something like what is shown [here](https://www.tensorflow.org/guide/tensorboard_histograms). – jdehesa Sep 05 '18 at 11:20
  • ok I got it. Just one more query. I see there are three different weights, viz `w_f_diag`,`w_i_diag` and`w_o_diag`. Which of these weights should be considered for the visualization so that I can get a firm decision making, whether my model is going well or not? – Jaffer Wilson Sep 05 '18 at 11:27
  • Also, sir, I would like to know if I wanted to see whats going on in my LSTM model inside or behind the wall, is there anything I can get to see? Like a real time working , how the association are made within the model of my Multi cell RNN? Sir, kindly enlighten me, please. – Jaffer Wilson Sep 05 '18 at 11:29
  • @JafferWilson Well in this case you have 5 weights per layer really, the kernel, the bias and the three you mention, which are for the "peepholes" (if you don't set `use_peephole=True` they will not appear). The naming for these I suppose follows the convention on [Understanding LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) (see "Variants on Long Short Term Memory"). [Here](https://stackoverflow.com/questions/42315202/understanding-tensorboard-weight-histograms) is a nice discussion on how to use histograms to figure out whether your training is working. – jdehesa Sep 05 '18 at 11:35
  • @JafferWilson I'm not completely sure what you mean with your latest comment. Maybe you could use [`tf.summary.image`](https://www.tensorflow.org/api_docs/python/tf/summary/image) to visualize the sequence of `rnn_outputs`? Not sure that's what you mean, or whether it would necessarily show something useful... – jdehesa Sep 05 '18 at 11:39
  • I wanted to show how the values are getting associated within the LSTM. The rows and columns that I am providing, how they are correlating to each other in order to form the output. This will help me in understanding if I am going wrong somewhere. And then I can change the data to proper format. May be. – Jaffer Wilson Sep 05 '18 at 11:41
  • @JafferWilson I see. Well that is difficult. Towards the end of [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) there is something (I think) kind of like that for text (which is simpler because each sample is a single value, not a vector), and is similar to what I was saying (plotting the activation of a random output neuron for each letter). It shows some patterns, but is not very systematic, and requires an analysis that is "easy" for text but maybe not in general... – jdehesa Sep 05 '18 at 11:49
  • Ok sir. You were saying that we can plot `rnn_outputs` variable using `tf.summary.image `, right, kindy, let me know how we can do that,please. – Jaffer Wilson Sep 05 '18 at 11:57
  • @JafferWilson Maybe you can post that as a different question? Not because of the points, but it's a rather different topic for this answer and I don't think I could answer in a comment, and also someone else might find it useful or have better answers... – jdehesa Sep 05 '18 at 12:17
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/179469/discussion-between-jaffer-wilson-and-jdehesa). – Jaffer Wilson Sep 05 '18 at 12:29
  • Here is the question: https://stackoverflow.com/questions/52185422/displaying-rnn-using-tf-summary-image-give-error-in-python-tensorflow, please let me know what I have mistaken. – Jaffer Wilson Sep 05 '18 at 12:42