Concatenate additional features after LSTM layer for Time Series Forecasting

Question

I have the following dataset:

                Feature 1   Feature 2 ...  Feature to Predict
2015-01-01         1000          8                 12
2015-01-02         1200          2                 22
2015-01-03         4000          4                 51
2015-01-04         2000          8                 33
2015-01-05         1000          5                 14

I want to predict the last feature ("Feature to predict") at time t + 1 using the previous n timestamps. To do this I used a multivariate LSTM that is trained using the data from t-n to t.

The fact is that I also have the possibility to obtain the other features ( Feature 1, Feature 2 ...) for the time 't+1' that I want to predict.

What I would like to do is to add these additional features after the LSTM layers and before the Dense layers and use them in the prediction of my 'Feauture to Predict'.

Right now my code, without the additional features but only with the 't-n' to 't' features, looks like this:

mdl = Sequential()
# create and fit the LSTM network
mdl.addLSTM(neuronsl1,activation = 'tanh' ,return_sequences=True, input_shape=(lags,n_features))
mdl.add(Dropout(0.2))
mdl.addLSTM(neuronsl2,activation = 'tanh' , input_shape=(lags,n_features))
mdl.add(Dropout(0.2))

--------->>> At this point i would like to add the additional features at time 't + 1'

mdl.add(Dense(neuronsl3))
mdl.add(Dense(neuronsl4))
mdl.add(Dense(1))

Any suggestions on how to do that?

Answer 1

I think I misunderstood your question. You have mentioned that you want to:

obtain the other features ( Feature 1, Feature 2 ...) for the time 't+1' that I want to predict.

And you have "Feature 1" and "Feature 2" in your dataset. But in the title you have mentioned "concatenate additional features". So, if you want to predict not only "Feature to Predict" but also "Feature 1", "Feature 2", etc. at timestep t+1, then:

• Just setting the number of units in the last Dense layer to the number of features you want to predict would achieve what you want. Plus if you want to get only the timestep t+1 then you need to set the return_sequences=False (which is the default case, of course) in the last LSTM layer. That's because the Dense layer is applied on the last axis and not on the whole data at once.

However, if you would like to concatenate the output of last LSTM layer with additional features (which you need to provide it as inputs of the model), you need to use Keras functional API and use concatenate function (or equivalent Concatenate layer):

mdl_input1 = Input(shape=(lags,n_features))

x = LSTM(neuronsl1, activation='tanh', return_sequences=True)(mdl_input1)
x = Dropout(0.2)(x)
x = LSTM(neuronsl2, activation='tanh')(x)
x = Dropout(0.2)(x)

mdl_input2 = Input(shape=(shape_of_features))

concat = concatenate([x, mdl_input2])

x = Dense(neuronsl3)(x)
x = Dense(neuronsl4)(x)
output = Dense(1)(x)

model = Model([mdl_input1, mdl_input2], output)

# compile the model ...

model.fit([input_array_for_lstm, input_array_additional_features], y_train, ...)