16

I am a newbie to LSTM and RNN as a whole, I've been racking my brain to understand what exactly is a timestep. I would really appreciate an intuitive explanation to this

Steven Wang
  • 335
  • 1
  • 2
  • 7

1 Answers1

20

Let's start with a great image from Chris Olah's blog (a highly recommended read btw):

enter image description here

In a recurrent neural network you have multiple repetitions of the same cell. The way inference goes is - you take some input (x0), pass it through the cell to get some output1(depicted with black arrow to the right on the picture), then pass output1 as input(possibly adding some more input components - x1 on the image) to the same cell, producing new output output2, pass that again as input to the same cell(again with possibly additional input component x2), producing output3 and so on.

A time step is a single occurrence of the cell - e.g. on the first time step you produce output1, h0, on the second time step you produce output2 and so on.

Ivaylo Strandjev
  • 69,226
  • 18
  • 123
  • 176
  • Thanks alot for the comment! I have a follow up question. If my code says timestep=40, does that mean that I start from the 40th timestep or I go upto the 40th timestep from the 1st timestep – Steven Wang Jan 17 '19 at 13:14
  • it is impossible to answer this without additional context. It seems this is a variable in your code but could refer to either I guess – Ivaylo Strandjev Jan 17 '19 at 13:17
  • The time timesteps were sliced between 10-40 and the 40th timestep was used. I have attached a link to the code for the LSTM model used. https://imgur.com/gallery/pejFKnY – Steven Wang Jan 17 '19 at 13:48
  • 2
    I believe there are two different `timestep` variables in action here. When you train a recurrent model you typically unroll it for a fixed number of steps and backpropagate, I believe this is the timestep in build_model. The others to my understanding and from this limited code snippet look like referring to the current time step – Ivaylo Strandjev Jan 17 '19 at 13:55
  • Thanks alot Ivaylo! what you've assumed checks out. really appreciate the help figuring this out! – Steven Wang Jan 17 '19 at 14:11