5

I have been working on LSTM for timeseries forecasting by using tensorflow. Now, i want to try sequence to sequence (seq2seq). In the official site there is a tutorial which shows NMT with embeddings . So, how can I use this new seq2seq module without embeddings? (directly using time series "sequences").

# 1. Encoder
encoder_cell = tf.contrib.rnn.BasicLSTMCell(LSTM_SIZE)
encoder_outputs, encoder_state = tf.nn.static_rnn(
  encoder_cell,
  x,
  dtype=tf.float32)

# Decoder
decoder_cell = tf.nn.rnn_cell.BasicLSTMCell(LSTM_SIZE)


helper = tf.contrib.seq2seq.TrainingHelper(
    decoder_emb_inp, decoder_lengths, time_major=True)


decoder = tf.contrib.seq2seq.BasicDecoder(
  decoder_cell, helper, encoder_state)

# Dynamic decoding
outputs, _ = tf.contrib.seq2seq.dynamic_decode(decoder)
outputs = outputs[-1]

# output is result of linear activation of last layer of RNN
weight = tf.Variable(tf.random_normal([LSTM_SIZE, N_OUTPUTS]))
bias = tf.Variable(tf.random_normal([N_OUTPUTS]))
predictions = tf.matmul(outputs, weight) + bias

What should be the args for TrainingHelper() if I use input_seq=x and output_seq=label?

decoder_emb_inp ??? decoder_lengths ???

Where input_seq are the first 8 point of the sequence, and output_seq are the last 2 point of the sequence. Thanks on advance!

dnovai
  • 137
  • 1
  • 8
  • Did you by any chance find an answer to this? I am following the same tutorial and want implement it for timeseries data – MrfksIV Jun 25 '18 at 12:50
  • Hi MrfksIV! I just found this tutorial [link](https://github.com/aaxwaz/Multivariate-Time-Series-forecast-using-seq2seq-in-TensorFlow/blob/master/time_series_seq2seq.ipynb) , but it does not use TensorFlow API which does not allow to use ML Engine (scalable training/ and deployment on GCP). I hope that this tutorial helps you to figure out how to construct a seq2seq for timeseries. If you figure out how to extend to TensorFlow API, please let me know. – dnovai Jun 25 '18 at 16:08
  • Will have a look at this tomorrow and let you know if I find anything. Thanks! – MrfksIV Jun 25 '18 at 16:36
  • I have found [this](https://stackoverflow.com/questions/44639647/tensorflow-1-2-how-to-setup-time-series-prediction-at-inference-time-using-seq2s) which helped me build exactly what I wanted. I think it will solve your problem as well! – MrfksIV Jun 26 '18 at 09:22
  • Did you get good results? I mean there is a better performance in comparison with just use LSTM architecture. Let me know if you get good results in order to coding to my use case. – dnovai Jun 26 '18 at 15:15
  • The validation error decreased from 2.4 to 0.03 so it seems that it is worth the extra effort. Note that the training time also increases substantially though. – MrfksIV Jun 26 '18 at 20:45
  • Thank you for your feedback, I will give it a try to my use case. – dnovai Jun 27 '18 at 15:25

1 Answers1

5

I got it to work for no embedding using a very rudimentary InferenceHelper:

inference_helper = tf.contrib.seq2seq.InferenceHelper(
        sample_fn=lambda outputs: outputs,
        sample_shape=[dim],
        sample_dtype=dtypes.float32,
        start_inputs=start_tokens,
        end_fn=lambda sample_ids: False)

My inputs are floats with the shape [batch_size, time, dim]. For the example below dim would be 1, but this can easily be extended to more dimensions. Here's the relevant part of the code:

projection_layer = tf.layers.Dense(
    units=1,  # = dim
    kernel_initializer=tf.truncated_normal_initializer(
        mean=0.0, stddev=0.1))

# Training Decoder
training_decoder_output = None
with tf.variable_scope("decode"):
    # output_data doesn't exist during prediction phase.
    if output_data is not None:
        # Prepend the "go" token
        go_tokens = tf.constant(go_token, shape=[batch_size, 1, 1])
        dec_input = tf.concat([go_tokens, target_data], axis=1)

        # Helper for the training process.
        training_helper = tf.contrib.seq2seq.TrainingHelper(
            inputs=dec_input,
            sequence_length=[output_size] * batch_size)

        # Basic decoder
        training_decoder = tf.contrib.seq2seq.BasicDecoder(
            dec_cell, training_helper, enc_state, projection_layer)

        # Perform dynamic decoding using the decoder
        training_decoder_output = tf.contrib.seq2seq.dynamic_decode(
            training_decoder, impute_finished=True,
            maximum_iterations=output_size)[0]

# Inference Decoder
# Reuses the same parameters trained by the training process.
with tf.variable_scope("decode", reuse=tf.AUTO_REUSE):
    start_tokens = tf.constant(
        go_token, shape=[batch_size, 1])

    # The sample_ids are the actual output in this case (not dealing with any logits here).
    # My end_fn is always False because I'm working with a generator that will stop giving 
    # more data. You may extend the end_fn as you wish. E.g. you can append end_tokens 
    # and make end_fn be true when the sample_id is the end token.
    inference_helper = tf.contrib.seq2seq.InferenceHelper(
        sample_fn=lambda outputs: outputs,
        sample_shape=[1],  # again because dim=1
        sample_dtype=dtypes.float32,
        start_inputs=start_tokens,
        end_fn=lambda sample_ids: False)

    # Basic decoder
    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,
                                                        inference_helper,
                                                        enc_state,
                                                        projection_layer)

    # Perform dynamic decoding using the decoder
    inference_decoder_output = tf.contrib.seq2seq.dynamic_decode(
        inference_decoder, impute_finished=True,
        maximum_iterations=output_size)[0]

Have a look at this question. Also I found this tutorial to be very useful to understand seq2seq models, although it does use embeddings. So replace their GreedyEmbeddingHelper by an InferenceHelper like the one I posted above.

P.s. I posted the full code at https://github.com/Andreea-G/tensorflow_examples

Andreea-G
  • 51
  • 1
  • 3