0

While implementing LSTM on keras, Input one-hot encoding of shape.x(None, 78) is converted into shape.x(None, 1, 78) but not sure why this conversion is required in keras.

I am looking at below code sample for DL course:

reshapor = Reshape((1, 78))                        
LSTM_cell = LSTM(n_a, return_state = True)         
densor = Dense(n_values, activation='softmax')

"""
Tx: 30
n_a = 64
n_values = 78
"""

def djmodel(Tx, n_a, n_values):
"""
Implement the model

Arguments:
Tx -- length of the sequence in a corpus
n_a -- the number of activations used in our model
n_values -- number of unique values in the music data 

Returns:
model -- a keras model with the 

Read this for input shape and dim on keras:
https://stackoverflow.com/questions/44747343/keras-input-explanation-input-shape-units-batch-size-dim-etc

"""

# Define the input of your model with a shape 
X = Input(shape=(Tx, n_values))

# Define s0, initial hidden state for the decoder LSTM
a0 = Input(shape=(n_a,), name='a0')
c0 = Input(shape=(n_a,), name='c0')
a = a0
c = c0

### START CODE HERE ### 
# Step 1: Create empty list to append the outputs while you iterate (≈1 line)
outputs = []

# Step 2: Loop
for t in range(Tx):

    # Step 2.A: select the "t"th time step vector from X. 
    x = Lambda(lambda x : X[:,t,:]) (X)
    # Step 2.B: Use reshapor to reshape x to be (1, n_values) (≈1 line)
    print("x.shape:{}".format(x.shape))

    #x = reshapor(x)
    print("x.shape:{}".format(x.shape))


    # Step 2.C: Perform one step of the LSTM_cell
    a, _, c = LSTM_cell(x, initial_state = [a,c])
    # Step 2.D: Apply densor to the hidden state output of LSTM_Cell
    out = densor(a)
    # Step 2.E: add the output to "outputs"


    outputs.append(out)

# Step 3: Create model instance
model = Model(inputs = [X,a0,c0],outputs = outputs)

### END CODE HERE ###

return model

I wanted know why reshaping is done above code sample.

Thanks

Jay
  • 41
  • 7
  • It's not clear at all what your question is. May be provide some context, examples so that what your want to clarify is clear. – thushv89 Apr 08 '20 at 12:27

1 Answers1

0

The input shape is supposed to be

  • [n_samples, n_time_steps, n_features]

Number of samples refer usually to your batch size, the number of time steps refer to the number of observation included in a single train (or test) instance, for example the number of words in a single sentence if you're processing text, the number of features refers to the number of features for each time step.

So if you have a single observation represented with 78 one-hot features, then you need to reshape that observation into a vector of shape with 78 as third dimension. It is weird though that in your question what you have seems to be a single time step, which is not something for which you should use LSTM.

Edoardo Guerriero
  • 1,210
  • 7
  • 16
  • Thanks for your response, We are using 30 observation (Tx=30 time_steps) and each observation is having 78 features and number of training example is 60. But Ia i am not clear with your answer, why we need to reshape features which are represented in one-hot encoding form into vector of shape with 78 as third dimension. Can you please clarify this . – Jay Apr 08 '20 at 14:00
  • I'll make an example based on text cause I mainly work on nlp, So let's say I have some sentences composed with a combination of a total number of 5 words. I want to use single words as an input feature, so theoretically I have only 1 feature per time step (the word at that time step). If instead I represent with one hot encoding my sentences, I will end up having 5 features per word (total amount of words, or the size of my vocabulary), because each word would be represented with a vector of size [1,5]. – Edoardo Guerriero Apr 08 '20 at 15:02
  • Now, when I feed my sentences to Keras, the number of features must be specified (represented) in the third dimension, there is no hidden explanation for this, it is just because of how tensorflow and Keras were designed. – Edoardo Guerriero Apr 08 '20 at 15:03
  • Thanks Edoardo, It clarify my doubts – Jay Apr 09 '20 at 09:06