python joblib.load: Iterating over generator results in infinite loop

Question

Below is the code I use for the generator. It is modified from Saving and loading multiple objects in pickle file?. The file was created with joblib.dump, and consists (so far) of a single machine learning model.

import pickle
from sklearn.externals import joblib

filename = 'ML_models.pkl'
def load(filename):
    with open(filename, "rb") as f:
        while True:
            try:
                yield joblib.load(f)
            except EOFError:
                break

Next I test the generator as follows:

models = load(filename)
for model in models:
    print model

Unfortunately, this prints the model repeatedly until I interrupt the kernel. What is the right way to signal EOF?

score 1 · Accepted Answer · answered Jan 16 '18 at 16:29

1

According to the documentation, joblib.load returns the full object.

def load(filename):
    with open(filename, "rb") as f:
        return joblib.load(f)

answered Jan 16 '18 at 16:29

Helio Santos

6,606
3
25
31

It looks like I should store my models e.g., in a dict or list, then dump that to file as a single object. – JGrossman Jan 16 '18 at 16:49
1

Indeed, it works if I dump multiple objects stored in a dict. – JGrossman Jan 16 '18 at 17:26

python joblib.load: Iterating over generator results in infinite loop

1 Answers1