3

I am trying to write a generator for my Keras lstm model. To use it with fit_generator method. My first question is what should my generator return? A batch? A sequence? Example in Keras documentation returns x,y for each data entry, but what if my data is sequential? And I want to split it into batches?

Here is the python method that creates a batch for a given input

def get_batch(data, batch_num, batch_size, seq_length):
    i_start = batch_num*batch_size;
    batch_sequences = []
    batch_labels = []
    batch_chunk = data.iloc[i_start:(i_start+batch_size)+seq_length].values
    for i in range(0, batch_size):
        sequence = batch_chunk[(i_start+i):(i_start+i)+seq_length];
        label = data.iloc[(i_start+i)+seq_length].values;
        batch_labels.append(label)
        batch_sequences.append(sequence)
    return np.array(batch_sequences), np.array(batch_labels);

The output of this method for an input like this:

get_batch(data, batch_num=0, batch_size=2, seq_length=3):

Would be:

x = [
      [[1],[2],[3]],
      [[2],[3],[4]]
    ]

Here is how I imagine my model:

model = Sequential()
model.add(LSTM(256, return_sequences=True, input_shape=(seq_length, num_features)))
model.add(Dropout(0.2))
model.add(LSTM(256))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')

My question is how can I translate my method into a generator?

nuric
  • 11,027
  • 3
  • 27
  • 42
Dmytro Kostyushko
  • 359
  • 3
  • 13

1 Answers1

3

Here is a solution that uses Sequence which acts like a generator in Keras:

class MySequence(Sequence):
  def __init__(self, num_batches):
    self.num_batches = num_batches

  def __len__(self):
    return self.num_batches # the length is the number of batches

  def __getitem__(self, batch_id):
    return get_batch(data, batch_id, self.batch_size, seq_length)

I think this is cleaner and doesn't modify your original function. Now you pass an instance of MySequence to model.fit_generator.

nuric
  • 11,027
  • 3
  • 27
  • 42
  • Thank, you this is just what I needed, this approach also allows me to use option `use_multiprocessing` in `fit_generator` to fetch batches asynchronicly. – Dmytro Kostyushko May 31 '18 at 12:20
  • Could you elaborate a bit more about the advantages of using a class with `__getitem__` instead of using a generator? Is it only that its thread-safe? – Markus May 29 '19 at 18:25
  • 1
    Yes, pretty much: *Sequence are a safer way to do multiprocessing. This structure guarantees that the network will only train once on each sample per epoch which is not the case with generators.* from [doc](https://keras.io/utils/#sequence). – nuric May 29 '19 at 19:55
  • Thank you @nuric! – Markus Jun 03 '19 at 18:07