Unable to replicate probability scores from persisted XGBClassifier despite setting random state and seed

Question

I am trying to reuse a saved XGBClassifer that after training makes the following predictions via .predict_proba() method:

ID -- scores

1 -- 0.072253475

2 -- 0.165827038

3 -- 0.098182471

4 -- 0.148302316

However, after reloading the object after either pickling it or using Skleans Joblib module, the predictions are totally off even after using the exact same test set:

ID -- scores

1 -- 0.46986327

2 -- 0.63513994

3 -- 0.45958066

4 -- 0.8958819

This is the classifier:

XGBClassifier(base_score=0.5, booster='gbtree',colsample_bylevel=1, 
colsample_bytree=0.8, gamma=1, learning_rate=0.01, max_delta_step=0,
   max_depth=4, min_child_weight=1, missing=nan, n_estimators=1500,
   n_jobs=-1, nthread=None, objective='binary:logistic',
   random_state=777, reg_alpha=2, reg_lambda=1,
   scale_pos_weight=0.971637216356233, seed=777, silent=True,
   subsample=0.6, verbose=2)

I pickle the object using using two different methods, Joblib provided in sklearn package and the standard pickle.dump using these functions:

def save_as_pickled_object(obj, filepath):
    import pickle
    import os
    import sys
    """
    This is a defensive way to write pickle.write, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    """
    Adding protocol = 4 as an argument to pickle.dumps because it allows for seralizing data greater than 4GB
    reference link: https://stackoverflow.com/questions/29704139/pickle-in-python3-doesnt-work-for-large-data-saving
    """    
    bytes_out = pickle.dumps(obj, protocol=4)
    n_bytes = sys.getsizeof(bytes_out)
    with open(filepath, 'wb') as f_out:
        for idx in range(0, n_bytes, max_bytes):
            f_out.write(bytes_out[idx:idx+max_bytes])

def try_to_load_as_pickled_object_or_None(filepath):
    import pickle
    import os
    import sys
    """
    This is a defensive way to write pickle.load, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    try:
        input_size = os.path.getsize(filepath)
        bytes_in = bytearray(0)
        with open(filepath, 'rb') as f_in:
            for _ in range(0, input_size, max_bytes):
                bytes_in += f_in.read(max_bytes)
        obj = pickle.loads(bytes_in)
    except:
        return None
    return obj

Regardless of how I save the file (joblib or pickle) the results are the same; that is, predict proba scores are totally different then when I use the method on the XGBClassifier object immediately after training.

On another note, On the same kernal, I am doing the same thing using an SGDClassifier, for which I am not experiencing this same problem.

One odd thing that I noticed is that if I save the classifier after training it, and then load it within the same kernal session (Using Jupyterlab), the probabilities are the same. However, if I restart the kernal, and load the object via either one of the two methods described above, then I no longer get the same probabilities.

My expectation is that I should be getting the same or nearly identical probabilities using the predict proba method on XGBClassifier.

Thanks

score 0 · Answer 1 · answered Apr 07 '19 at 11:58

I guess, there is either something funny in the way how you dump and read the model, or it is a feature of the version of xgboost, that you use.

I can fully reproduce predicted probabilities using simple loading of a persisted XGB model with the following code in a notebook (repeating "Load the model" section and initial imports after a kernel restart)

import os
import numpy as np
import pandas as pd
import pickle
import joblib
import xgboost as xgb


## Training a model
np.random.seed(312)
train_X = np.random.random((10000,10))
train_y = np.random.randint(0,2, train_X.shape[0])
val_X = np.random.random((10000,10))
val_y = np.random.randint(0,2, train_y.shape[0])

xgb_model_mpg = xgb.XGBClassifier(max_depth= 3)
_ = xgb_model_mpg.fit(train_X, train_y)
print(xgb_model_mpg.predict_proba(val_X))


## Save the model
with open('m.pkl', 'wb') as fout:
    pickle.dump(xgb_model_mpg, fout)

joblib_dump(xgb_model_mpg, 'm.jlib')


## Load the model
m_jlb = joblib.load('m.jlib')
m_pkl = pickle.load( open( "m.pkl", "rb" ) )

print(m_jlb.predict_proba(val_X))
print(m_pkl.predict_proba(val_X))

I use xgboost 0.71 and joblib 0.12.4 in a regular jupyter notebook in python 3.5.5

Unable to replicate probability scores from persisted XGBClassifier despite setting random state and seed

1 Answers1