Is it possible to making LSTM model with 4 dimension shape of data?

Question

Hellow, wizards.
I have time series data including sevaral days.
I try to predict a grade of tomorrow, which is range from 0 to 100.
And I assume that this grade depends on 3 time-series independent features.

Every each day is a unit and has a grade, but each unit(day) has different length of time series data.
Because each day has lots of data length, I rolled window, which size is 1080 and stride is 180, on each day.

So I tried to transform it into (# of unit(sample), # of window, windowsize, features).
Finally I get (450, # window(not fixed), 1080, 3).

450 days, Each day has same length window(1080) and features(3), but Number of windows are different.
I condisered zero-padding, but I heard that I can use dynamic(?) LSTM with differnent length of input.

The problem is that shape of input of Keras LSTM must be 3 dimension.
So I'm confused how I should fit this shape to LSTM model.
My goal is that when I put a new day, which shape is (# of window, 1080, 3) into LSTM model, I want it to predict a grade.

Could you give me any suggestions or advices? Thank you.

Answer 1

The answer to your question is, unfortunately, no. The input shape of an LSTM layer in Keras has to be (batch_size, timesteps, features). No 4th dimension is possible.

However, as you seem to be following the sliding window approach, I can tell you that the usual way is to have (# of windows) in the batch dimension. In your case that would mean your input shape would become (450 * #window, 1080, 3).

Then, for a prediction you would have to use only the last window of a day, you cannot use the entire day.

Note, that this is in general one huge limitation of the sliding window approach. If a good prediction needs data of the entire day instead of just one window of that day, this approach is not gonna work well.

You can still give it a try and play with different window sizes. If it doesn't work, I'd suggest you discard this approach and use the data of one day directly, instead of dividing it into windows first. So, (450, #(data of a day), 3) would be your input shape. It is correct what you have heard: You can have variable length on the timestep axis. To achieve this, you have to declare input_shape=(None, 3) in the first layer.

The only downside of this is that you cannot put together all of your data as one numpy array because numpy does not work with different sizes. In other words, instead of one numpy array of shape (450, #(data of a day), 3) you will need 450 numpy arrays of shape (1, #(data of a day), 3) where #(data of a day) can be different each time. Unfortunately, model.fit() cannot be used for this as it expects the training data to be one single numpy array. Instead, you could use model.fit_generator and pass it a generator that yields one of these 450 small arrays on each call. Have a look at the docs for examples on how to use this. Note that your training may slow down due to batch_size=1.

If this turns out to be an issue, LSTM layers in Keras support masking. For this, simply use a Masking layer as the very first one in your model and declare one value (usually 0.0, but can be any number) to be masked. Then you can create input tensors of shape (batch_size, #max_day_data, 3) where all days that have less than #max_day_data data must be right-padded with the value that will be masked.

For further reading I highly recommend this answer on LSTMs. Maybe this will inspire you for even different approaches you could try. Especially using stateful might be an option in case you have too much data for one day that doesn't fit into your memory.