Understanding Seq2Seq model

Question

Here is my understanding of a basic Sequence to Sequence LSTMs. Suppose we are tackling a question-answer setting.

You have two set of LSTMs (green and blue below). Each set respectively sharing weights (i.e. each of the 4 green cells have the same weights and similarly with the blue cells). The first is a many to one LSTM, which summarises the question at the last hidden layer/ cell memory.

The second set (blue) is a Many to Many LSTM which has different weights to the first set of LSTMs. The input is simply the answer sentence while the output is the same sentence shifted by one.

The question is two fold: 1. Are we passing the last hidden state only to the blue LSTMs as the initial hidden state. Or is it last hidden state and cell memory. 2. Is there a way to set the initial hiddden state and cell memory in Keras or Tensorflow? If so reference?

(image taken from suriyadeepan.github.io)

Answer 1

1. Are we passing the last hidden state only to the blue LSTMs as the initial hidden state. Or is it last hidden state and cell memory.

Both hidden state h and cell memory c are passed to the decoder.

TensorFlow

In seq2seq source code, you can find the following code in basic_rnn_seq2seq():

_, enc_state = rnn.static_rnn(enc_cell, encoder_inputs, dtype=dtype)
return rnn_decoder(decoder_inputs, enc_state, cell)

If you use an LSTMCell, the returned enc_state from the encoder will be a tuple (c, h). As you can see, the tuple is passed directly to the decoder.

Keras

In Keras, the "state" defined for an LSTMCell is also a tuple (h, c) (note that the order is different from TF). In LSTMCell.call(), you can find:

    h_tm1 = states[0]
    c_tm1 = states[1]

To get the states returned from an LSTM layer, you can specify return_state=True. The returned value is a tuple (o, h, c). The tensor o is the output of this layer, which will be equal to h unless you specify return_sequences=True.

2. Is there a way to set the initial hiddden state and cell memory in Keras or Tensorflow? If so reference?

###TensorFlow### Just provide the initial state to an LSTMCell when calling it. For example, in the official RNN tutorial:

lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)
...
    output, state = lstm(current_batch_of_words, state)

There's also an initial_state argument for functions such as tf.nn.static_rnn. If you use the seq2seq module, provide the states to rnn_decoder as have been shown in the code for question 1.

###Keras###

Use the keyword argument initial_state in the LSTM function call.

out = LSTM(32)(input_tensor, initial_state=(h, c))

You can actually find this usage on the official documentation:

###Note on specifying the initial state of RNNs###

You can specify the initial state of RNN layers symbolically by calling them with the keyword argument initial_state. The value of initial_state should be a tensor or list of tensors representing the initial state of the RNN layer.

---

EDIT:

There's now an example script in Keras (lstm_seq2seq.py) showing how to implement basic seq2seq in Keras. How to make prediction after training a seq2seq model is also covered in this script.

Answer 2

(Edit: this answer is incomplete and hasn't considered actual possibilities of state transfering. See the accepted answer).

From a Keras point of view, that picture has only two layers.

• The green group is one LSTM layer.
• The blue group is another LSTM layer.

There isn't any communication between green and blue other than passing the outputs. So, the answer for 1 is:

1. Only the thought vector (which is the actual output of the layer) is passed to the other layer.

Memory and state (not sure if these are two different entities) are totally contained inside a single layer and are not initially intended to be seen or shared with any other layer.

Each individual block in that image is totally invisible in keras. They are considered "time steps", something that only appears in the shape of the input data. It's rarely important to worry about them (unless for very advanced usages).

In keras, it's like this:

Easily, you have access only to the external arrows (including "thought vector").
But having access to each step (each individual green block in your picture) is not an exposed thing. So...

2. Passing the states from one layer to the other is also not expected in Keras. You will probably have to hack things. (See this: https://github.com/fchollet/keras/issues/2995)

But considering a thought vector big enough, you could say it will learn a way to carry what is important in itself.

The only notion you have from the steps is:

• You have to input things shaped like (sentences, length, wordIdFeatures)

The steps will be performed considering that each slice in the length dimension is an input to each green block.

You may choose to have a single output (sentences, cells), for which you completely lose track of steps. Or...

• Outputs like (sentences, length, cells), from which you know the output of each block through the length dimension.

One to many or many to many?

Now, the first layer is many to one (but nothing prevents it from being many to many too if you want).

But the second... that's complicated.

• If the thought vector was made by a many to one. You will have to manage a way of creating a one to many. (That's not trivial in keras, but you could think of repeating the thought vector for the expected length, making it be the input to all steps. Or maybe fill an entire sequence with zeros or ones, keeping only the first element as the thought vector)
• If the thought vector was made by a many to many, you can take advantage of this and keep an easy many to many, if you're willing to accept that the output has exactly the same number of steps as the input.

Keras doesn't have a ready solution for 1 to many cases. (From a single input predict a whole sequence).