How to stack multiple lstm in keras without return_sequences?

Question

here is my code

model = Sequential()
model.add(LSTM(256, input_shape=(look_back, x_train.shape[1])))
model.add(Dropout(0.3))

model.add(LSTM(256, input_shape=(look_back, x_train.shape[1])))
model.add(Dropout(0.3))

then occur ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=2

but I can't use return_sequence

because of here

So, How to stack multiple lstm in keras without return_sequences?

Answer 1

model = Sequential()
model.add(LSTM(256,return_sequences=True, input_shape=(10, 20)))
model.add(LSTM(128,return_sequences=True))
model.add(LSTM(64,return_sequences=True))
model.add(LSTM(32,return_sequences=True))
model.add(LSTM(16))
model.add(Dropout(0.3))
model.add(Dense(1, activation='linear'))

Lets see what exactly a return_Sequence does:

with return_sequence set to True, every LSTM unwrapping will will return an output. With return_sequence set to False, only the LSTMS last unwrapping will be returned.

The Problem

Now your problem is give a sequence of numbers predict the next number. If you plan to use stacked LSTMS how will you use it ? You will stack up one LSTM over other like a block of brick. So Every LSTM cell above will receive its inputs which is the LSTM output below it. However for the last layer you are only interested in the last state since you want to make a single prediction of the next value. So for the last LSTM you set return_sequence=False

And about your linked question, you can use that answer only if you have single LSTM cell, but if you want to stack up then this is what you have to follow.

Check model.summary() for understanding the model architecture.

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_9 (LSTM)                (None, 10, 256)           283648    
_________________________________________________________________
lstm_10 (LSTM)               (None, 10, 128)           197120    
_________________________________________________________________
lstm_11 (LSTM)               (None, 10, 64)            49408     
_________________________________________________________________
lstm_12 (LSTM)               (None, 10, 32)            12416     
_________________________________________________________________
lstm_13 (LSTM)               (None, 16)                3136      
_________________________________________________________________
dropout_8 (Dropout)          (None, 16)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 17        
=================================================================
Total params: 545,745
Trainable params: 545,745
Non-trainable params: 0
_________________________________________________________________

Answer 2

What is the reason behind not to use return sequence? I am not sure whether this code will give you a correct answer. But tried it and let me know if this is work or not.

model = Sequential()
model.add(Dense(32, input_shape=(look_back, x_train.shape[1])))
model.add(Dropout(0.3))


model.add(LSTM(16))
model.add(Dropout(0.3))


model.add(Dense(1))

Thank you