2

I trained a logistic regression model on textual data and saved the model using pickle. But for testing when I try to load the model I got the error mentioned in the title while executing the following line:

model = pickle.load(open("sentiment.model", "rb"))

Following is the code used for saving the model:

import pickle

print("[INFO] saving Model...")
f = open('sentiment.model', "wb")

# first I saved the best_estimator_
f.write(pickle.dumps(gs_lr_tfidf.best_estimator_))

# but again I saved the model completely without mentioning any attribute i.e: 
# f.write(pickle.dumps(gs_lr_tfidf))
# but none of them helped and I got the same error

f.close()
print("[INFO] Model saved!")

This error doesn't show up when I load the model in the same notebook just after finishing the training process (in the same runtime). But this error occurs when I try to load the model separately in different runtime even if the model loader code is the same. Why this is happening?

hafiz031
  • 2,236
  • 3
  • 26
  • 48
  • Does this answer your question? [Unable to load files using pickle and multiple modules](https://stackoverflow.com/questions/27732354/unable-to-load-files-using-pickle-and-multiple-modules) – jkr Sep 07 '20 at 15:28
  • @jakub I saw this question but didn't understand clearly what to do. – hafiz031 Sep 07 '20 at 15:29
  • 1
    Can you please include a minimal example that reproduces this behavior? My guess is that you will have to import some packages before loading the model. – jkr Sep 07 '20 at 15:41
  • 1
    @jakub this code itself produces this behavior. I tried putting all the code used for training (including all imports, grid search parameters, pipeline information and everything except the `fit()` function [as I am not training it again]) before loading model and this error gone. Although I didn't explicitly pass anything to this `pickle.load()` function. – hafiz031 Sep 07 '20 at 15:53
  • 1
    Does this help? https://stackoverflow.com/questions/50465106/attributeerror-when-reading-a-pickle-file – Kim Tang Sep 07 '20 at 23:39
  • @hafiz031 : You should write an answer for the above question - your **imports** idea worked for me, I did the imports and for me also the error is gone. – aspiring1 Feb 08 '22 at 05:10

1 Answers1

1

I think the problems is from the behaviour of pickle, as what @hafiz031 said, it's normal when run the same code in the file. So short answer is you need to import tokenizer(from whatever lib you use) before you load the model

For people who know chinese, you can go to this CSDN link for more info.

For people who don't know chinese, sorry for my bad English and I'll try my best to explain.

The documentation says:

pickle.loads(data, /, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=None)

Return the reconstituted object hierarchy of the pickled representation data of an object. data must be a bytes-like object.

There is an implicit requirement if you use pickle.loads, the object hierarchy must be declared before you load it. Intuitively you can think as you bring USD to north pole and you want to exchange USD to fish with a penguin. As they don't have the concept what is money, they won't make the deal. Same as pickle, if you haven't import tokenizer before, after pickle loads the bytes back to tokenizer, they don't know what is 'tokenizer' and return error to you. Thats why your code works in training file but fail when you loads the model in a different file.

in my case, I just import an extra lib.

# import your own lib
import pickle

import nltk.tokenizer
import genism
import sklearn
#...

model = pickle.load(open("sentiment.model", "rb"))
#model.predict()
ChrisGPT was on strike
  • 127,765
  • 105
  • 273
  • 257
c yipkwun
  • 31
  • 3