I use compressed pickle to save the results from sklearn gridsearch using the following code.
import pickle
import bz2
from sklearn.model_selection import RandomizedSearchCV
search = RandomizedSearchCV(estimator, param_distributions=param_dist,
cv = cv,
n_jobs = 6
)
with bz2.open('gridsearch.bz2', 'wb') as f:
pickle.dump(search, f)
Then I use the following code to get the search object.
with bz2.open('gridsearch.bz2', 'rb') as f:
search = pickle.load(f)
The pickle load works when I run in virtual environment in terminal but it report the following error if I run the code in jupyter lab. I checked the version of pickle. Both the virtual environment and jupyter lab use pickle 4.0.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-11-06b38588a6ad> in <module>
---> 23 with bz2.open('gridsearch.bz2', 'rb') as f:
24 search = pickle.load(f)
~/.pyenv/versions/3.7.4/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py in __setstate__(self, state)
623
624 def __setstate__(self, state):
--> 625 ctor_param, r = state
626 self.__init__(**ctor_param)
627 self._random_state = r
ValueError: too many values to unpack (expected 2)
BTW, I have also tried to use .pickle file instead of bz2. This problem still occurs. Any comment and suggestions would be appreciated.