1

I am trying to process data and use it later for training a model in keras. However, I am running out of memory while returning 4 arrays at the same time to the main function. The code looks like this:

def prepare_data:
    all_3d_data = []
    for j in range(all_3d_data):
        ...
        X_train.append(all_3d_data)
    del all_3d_data

There are four similar loops in each of which one array is filled with some data. These 4 arrays are returned to the main function at once:

    return X_train, Y_train, X_test, Y_test

In the main function when it is called:

if __name__ == "__main__":
    X_train, Y_train, X_test,Y_test = prepare_data()

An error raises:

File "C:...\tensorflow\lib\site-packages\numpy\core\numeric.py", line 501, in asarray return array(a, dtype, copy=False, order=order)

MemoryError

This error doesn't happen when I am using smaller data. How can I fix this? Should I use batch in Keras?

aschipfl
  • 33,626
  • 12
  • 54
  • 99
Hadi GhahremanNezhad
  • 2,377
  • 5
  • 29
  • 58
  • 1
    Try to use generator to train the model. You can check [this](https://stackoverflow.com/questions/46493419/use-a-generator-for-keras-model-fit-generator). And to fit the data use [`model.fit_generator`](https://keras.io/models/sequential/#fit_generator) – Krunal V Apr 30 '19 at 14:06
  • @kruxx thank you for the hint. But I am still not sure how to do this. Should I add a while loop at the beginning of the `prepare_data` function to check with `batch_size` and use `yield` instead of `return`? Does it continue from the next batch at each run? – Hadi GhahremanNezhad Apr 30 '19 at 23:15
  • 1
    Yes! you need to use `yield` instead of `return`. Please have look on [this](https://www.pyimagesearch.com/2018/12/24/how-to-use-keras-fit-and-fit_generator-a-hands-on-tutorial/), it include detailed explanation. – Krunal V May 02 '19 at 08:21

0 Answers0