Model and shapes
Since these are sequences in sequences, you need to use your data in a different format.
Although you could just go like (machines, days, 360) and simply treat the 360 as features (that could work up to some point), for a robust model (then maybe there is a speed problem) you'd need to treat both things as sequences.
Then I'd go with data like (machines, days, 360, 1) and two levels of recurrency.
Our models input_shape then would be (None, 360, 1)
Model case 1 - Only day recurrency
Data shape: (machines, days, 360)
Apply some normalization to the data.
Here, an example, but models can be flexible as you can add more layers, try convolutions, etc:
inputs = Input((None, 360)) #(m, d, 360)
outs = LSTM(some_units, return_sequences=False,
stateful=depends_on_training_approach)(inputs) #(m, some_units)
outs = Dense(360, activation=depends_on_your_normalization)(outs) #(m, 360)
outs = Reshape((1,360)) #(m, 1, 360)
#this reshape is not necessary if using the "shifted" approach - see time windows below
#it would then be (m, d, 360)
model = Model(inputs, outs)
Depending on the complexity of the intra-daily sequences, they could get well predicted with this, but if they evolve in a complex way, then the next model would be a little better.
Always remember that you can create more layers and explore things to increase the capability of this model, this is only a tiny example
Model case 2 - Two level recurrency
Data shape: (machines, days, 360, 1)
Apply some normalization to the data.
There are so many many ways to experiment on how to do this, but here is a simple one.
inputs = Input((None, 360, 1)) #(m, d, 360, 1)
#branch 1
inner_average = TimeDistributed(
Bidirectional(
LSTM(units1, return_sequences=True, stateful=False),
merge_mode='ave'
)
)(inputs) #(m, d, 360, units1)
inner_average = Lambda(lambda x: K.mean(x, axis=1))(inner_average) #(m, 360, units1)
#branch 2
inner_seq = TimeDistributed(
LSTM(some_units, return_sequences=False, stateful=False)
)(inputs) #may be Bidirectional too
#shape (m, d, some_units)
outer_seq = LSTM(other_units, return_sequences = False,
stateful=depends_on_training_approach)(inner_seq) #(m, other_units)
outer_seq = Dense(few_units * 360, activation = 'tanh')(outer_seq) #(m, few_units * 360)
#activation = same as inner_average
outer_seq = Reshape((360,few_units))(outer_seq) #(m, 360, few_units)
#join branches
outputs = Concatenate()([inner_average, outer_seq]) #(m, 360, units1+few_units)
outputs = LSTM(units, return_sequences=True, stateful= False)(outputs) #(m, 360,units)
outputs = Dense(1, activation=depends_on_your_normalization)(outputs) #(m, 360, 1)
outputs = Reshape((1,360))(outputs) #(m, 1, 360) for training purposes
model = Model(inputs, outputs)
This is one attempt, I made an average of the days, but I could have made, instead of inner_average, something like:
#branch 1
daily_minutes = Permute((2,1,3))(inputs) #(m, 360, d, 1)
daily_minutes = TimeDistributed(
LSTM(units1, return_sequences=False,
stateful=depends_on_training_approach)
)(daily_minutes) #(m, 360, units1)
Many other ways of exploring the data are possible, this is a highly creative field. You could, for instance, use the daily_minutes approach right after the inner_average excluding the K.mean lambda layer.... you got the idea.
Time windows approach
Your approach sounds nice. Give one step to predict the next, give two steps to predic the third, give three steps to predict the fourth.
The models above are suited to this approach.
Keep in mind that very short inputs may be useless and may make your model worse. (Try to imagine how many steps would be reasonably enough for you to start predicting the next ones)
Preprocess your data and divide it in groups:
- group with length = 4 (for instance)
- group with length = 5
- ...
- group with length = 28
You will need a manual training loop where in each epoch you feed each of these groups (you can't feed different lenghts all together).
Another approach is, give all steps, make the model predict a shifted sequence like:
inputs = original_inputs[:, :-1] #exclude last training day outputs = original_inputs[:, 1:] #exclude first training day
For making the models above suited to this approach, you need return_sequences=True in every LSTM that uses the day dimension as steps (not the inner_seq). (The inner_average method will fail, and you will have to resort to the daily_minutes approach with return_sequences=True and another Permute((2,1,3)) right after.
Shapes would be:
- branch1 :
(m, d, 360, units1)
- branch2 :
(m, d, 360, few_units) - needs to adjust the Reshape for this
- The reshapes using 1 timestep will be unnecessary, the
days dimension will replace the 1.
- You may need to use
Lambda layers to reshape considering the batch size and variable number of days (if details are needed, please tell me)
Training and predicting
(Sorry for not having the time for detailing it now)
You then can follow the approaches mentioned here and here too, more complete with a few links. (Take care with the output shapes, though, in your question, we are always keeping the time step dimension, even though it may be 1)
The important points are:
- If you opt for
stateful=False:
- this means easy training with
fit (as long as you didn't use the "different lengths" approach);
- this also means you will need to build a new model with
stateful=True, copy the weights of the trained model;
- then you do the manual step by step prediction
- If you opt for
stateful=True from the beginning:
- this necessarily means manual training loop (using
train_on_batch for instance);
- this necessarily means you will need
model.reset_states() whenever you are going to present a batch whose sequences are not sequels of the last batch (every batch if your batches contain whole sequences).
- don't need to build a new model to manually predict, but manual prediction remains the same
