How to apply LSTM-autoencoder to variant-length time-series data?

Question

I read LSTM-autoencoder in this tutorial: https://blog.keras.io/building-autoencoders-in-keras.html, and paste the corresponding keras implementation below:

from keras.layers import Input, LSTM, RepeatVector
from keras.models import Model

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(latent_dim)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)
encoder = Model(inputs, encoded)

In this implementation, they fixed the input to be of shape (timesteps, input_dim), which means length of time-series data is fixed to be timesteps. If I remember correctly RNN/LSTM can handle time-series data of variable lengths and I am wondering if it is possible to modify the code above somehow to accept data of any length?

Thanks!

score 6 · Accepted Answer · answered Sep 29 '17 at 20:29

6

You can use shape=(None, input_dim)

But the RepeatVector will need some hacking taking dimensions directly from the input tensor. (The code works with tensorflow, not sure about theano)

import keras.backend as K

def repeat(x):

    stepMatrix = K.ones_like(x[0][:,:,:1]) #matrix with ones, shaped as (batch, steps, 1)
    latentMatrix = K.expand_dims(x[1],axis=1) #latent vars, shaped as (batch, 1, latent_dim)

    return K.batch_dot(stepMatrix,latentMatrix)


decoded = Lambda(repeat)([inputs,encoded])
decoded = LSTM(input_dim, return_sequences=True)(decoded)

answered Sep 29 '17 at 20:29

Daniel Möller

84,878
18
192
214

Thanks! Do you have any idea how to pass data of variant length into the autoencoder? I tried to convert a list of var-length arrays into array but failed. I tried to pass a list of var-length arrays directly to it, but I got error info saying `Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 arrays but instead got the following list of 3773 arrays: [array([[ 0.300544 , 0.251966 ],`. – username123 Sep 29 '17 at 21:13
https://stackoverflow.com/questions/46144191/keras-misinterprets-training-data-shape/46146146#46146146 – Daniel Möller Sep 29 '17 at 22:06
1

There is also the option to pad the arrays with dummy values so they all get the same size, and use `masking`. (I never used it, but you can google how to use masking on keras). – Daniel Möller Sep 29 '17 at 22:09
I noticed another problem, every time when a group of data with new shape are fed into the autoencoder (in the for loop), the error will suddenly increase and then decrease gradually. Does it mean the parameters for data with different length should be different? Is there a better way to properly "mix" data with different lengths such that training results do not depend much on the order in which data of different lengths are used? – username123 Sep 30 '17 at 00:53
1

To mix them, you can use padding + masking and shuffle (which is automatic in `fit`, but you can assure with `shuffle=True`). It's not unusual to have different losses for different sequences. It's not necessarily due to the length. But you should not train one sequence many times, but cycle sequences. One epoch only with each sequence (I suggest `train_on_batch`), then you repeat all sequences. – Daniel Möller Sep 30 '17 at 01:59
@DanielMöller what if I have a larger lstm (more hidden units) than my seq_len? How can I reshape the output of lstm to the shape of my required output ?https://stackoverflow.com/questions/51709005/sparse-lstm-autoencoder-in-keras – Birish Aug 07 '18 at 06:47

How to apply LSTM-autoencoder to variant-length time-series data?

1 Answers1

Linked