How to find the source of a MemoryError in Python?

Question

I'm running a hyperparameter optimization using Hyperopt for a Neural Network. While doing so, after some iterations, I get a MemoryError exception

So far, I tried clearing all variables after they had been used (assigning None or empty lists to them, is there a better way for this?) and printing all locals(), dirs() and globals() with their sizes, but those counts never increase and the sizes are quite small.

The structure looks like this:

def create_model(params):
    ## load data from temp files
    ## pre-process data accordingly
    ## Train NN with crossvalidation clearing Keras' session every time
    ## save stats and clean all variables (assigning None or empty lists to them)

def Optimize():
    for model in models: #I have multiple models
        ## load data
        ## save data to temp files
        trials = Trials()
        best_run = fmin(create_model,
                        space,
                        algo=tpe.suggest,
                        max_evals=100,
                        trials=trials)

After X number of iterations (sometimes it completes the firsts 100 and shifts to the second model) it throws a memory error. My guess is that some variables remain in memory and I'm not clearing them, but I wasn't able to detect them.

EDIT:

Traceback (most recent call last):
  File "Main.py", line 32, in <module>
    optimal = Optimize(training_sets)
  File "/home/User1/Optimizer/optimization2.py", line 394, in Optimize
    trials=trials)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 307, in fmin
    return_argmin=return_argmin,
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/base.py", line 635, in fmin
    return_argmin=return_argmin)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 320, in fmin
    rval.exhaust()
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 199, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.async)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 173, in run
    self.serial_evaluate()
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 92, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/base.py", line 840, in evaluate
    rval = self.fn(pyll_rval)
  File "/home/User1/Optimizer/optimization2.py", line 184, in create_model
    x_train, x_test = x[train_indices], x[val_indices]
MemoryError

Could you add this error with the full stack trace in your question? — gbajson, Apr 14 '19 at 18:24
https://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended — Alex Jadczak, Apr 15 '19 at 00:36
@AlexJadczak that was useful. I used `memory_profiler` and 90% of the memory is generated in the line that calls to `fmin` — bleand, Apr 15 '19 at 19:53

score 2 · Accepted Answer · answered Apr 17 '19 at 05:48

It took me a couple of days to figure this out so I'll answer my own question to save whoever encounters this issue some time.

Usually, when using Hyperopt for Keras, the suggested return of the create_model function is something like this:

return {'loss': -acc, 'status': STATUS_OK, 'model': model}

But in large models, with many evaluations, you don't want to return and save in memory every model, all you need is the set of Hyperparameters that gave the lowest loss

By simply removing the model from the returned dict, the issue of memory increasing with each evaluation is resolved.

return {'loss': -acc, 'status': STATUS_OK}

How to find the source of a MemoryError in Python?

1 Answers1