LSTM Batches vs Timesteps

Question

I've followed the TensorFlow RNN tutorial to create a LSTM model. However, in the process, I've grown confused as to the difference, if any, between 'batches' and 'timesteps', and I'd appreciate help in clarifying this matter.

The tutorial code (see following) essentially creates 'batches' based on a designated number of steps:

with tf.variable_scope("RNN"):
      for time_step in range(num_steps):
        if time_step > 0: tf.get_variable_scope().reuse_variables()
        (cell_output, state) = cell(inputs[:, time_step, :], state)
        outputs.append(cell_output)

However, the following appears to do the same:

    for epoch in range(5):
        print('----- Epoch', epoch, '-----')
        total_loss = 0
        for i in range(inputs_cnt // BATCH_SIZE):
            inputs_batch = train_inputs[i * BATCH_SIZE: (i + 1) * BATCH_SIZE]
            orders_batch = train_orders[i * BATCH_SIZE: (i + 1) * BATCH_SIZE]
            feed_dict = {story: inputs_batch, order: orders_batch}

            logits, xent, loss = sess.run([...], feed_dict=feed_dict)

score 2 · Accepted Answer · answered Feb 02 '17 at 19:51

2

Assuming you are working with text, BATCH_SIZE would be the number of sentences that you are processing in parallel and num_steps would be the maximum number of words in any sentence. These are different dimensions of your input to the LSTM.

answered Feb 02 '17 at 19:51

Aaron

2,354
1
17
25

LSTM Batches vs Timesteps

1 Answers1

Linked