How to implement the time sequence shown as following

Question

The ST(spatial transform) has two input. the first one is Fi, which is fixed. The another is M, which is varied according to the output of last LSTM. The input of LTSM depends on the output of ST and the state of last LSTM.

Answer 1

The easiest way to do this is probably to write your own RNN cell. Another way is to use tf.raw_rnn. Check out this post or this excellent article.

Answer 2

Actually,I implement the network as following:

  def build_model(self):

    lstm_cell = tf.contrib.rnn.BasicLSTMCell(
        num_units=self.config.num_lstm_units, state_is_tuple=True, reuse=True)
    if self.mode == "train":
      lstm_cell = tf.contrib.rnn.DropoutWrapper(
          lstm_cell,
          input_keep_prob=self.config.lstm_dropout_keep_prob,
          output_keep_prob=self.config.lstm_dropout_keep_prob)

    with tf.variable_scope("lstm", initializer=self.initializer) as lstm_scope:

      zero_state = lstm_cell.zero_state(
          batch_size=self.image_embeddings.get_shape()[0], dtype=tf.float32)

      K = 5
      C = 80

      scores = tf.Variable(tf.random_normal(shape=[K, self.config.batch_size, C]), name="scores")

      M = tf.Variable(tf.random_normal(shape=[K+1, self.config.batch_size, 2, 3]), name="M")
      tf.assign(M[0], tf.convert_to_tensor([[1., 0., 0.], [0., 1., 0.]]))

      lstm_input_size = 14
      zk_size = 4096

      hidden = zero_state

      for k in range(0, K+1):
          # Allow the LSTM variables to be reused.
          if k > 0:
              lstm_scope.reuse_variables()
          f_k = spatial_transformer_network.spatial_transformer_network(self.image_embeddings, M[k])

          f_k = tf.nn.max_pool(f_k, [1,2,2,1], [1,1,1,1], padding='VALID')

          f_k = tf.layers.dense(tf.reshape(f_k, [self.config.batch_size, int(lstm_input_size * lstm_input_size / 4 * 512)]), 4096)

          lstm_outputs, hidden = lstm_cell(f_k, hidden)

          z_k = tf.layers.dense(hidden[0], zk_size, activation=tf.nn.relu)

          if k != 0:
              tf.assign(scores[k - 1], (tf.layers.dense(z_k, C)))

          if k != K:
              tf.assign(M[k + 1], (tf.reshape(tf.layers.dense(z_k, 6), [self.config.batch_size, 2, 3])))
              tf.assign(M[k + 1, :, 0, 1], (tf.convert_to_tensor(0.)))
              tf.assign(M[k + 1, :, 1, 0], (tf.convert_to_tensor(0.)))

But it throws error when it runs to

lstm_outputs, hidden = lstm_cell(f_k, hidden).

The error info is: ValueError: Variable lstm/basic_lstm_cell/kernel does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?

So what's the problem?