10

I am trying to fit XGBClassifier to my dataset after hyperparameter tuning using optuna and I keep getting this warning:

the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'

Below is my code:

#XGBC MODEL
model = XGBClassifier(random_state = 69)

cross_rfc_score = -1 * cross_val_score(model, train_x1, train_y,
                           cv = 5, n_jobs = -1, scoring = 'neg_mean_squared_error')
base_rfc_score = cross_rfc_score.mean()

But if I use Optuna and then fit the obtained parameters it gives me the warning. Below is the code:

def objective(trial):
    learning_rate = trial.suggest_float('learning_rate', 0.001, 0.01)
    n_estimators = trial.suggest_int('n_estimators', 10, 500)
    sub_sample = trial.suggest_float('sub_sample', 0.0, 1.0)
    max_depth = trial.suggest_int('max_depth', 1, 20)

    params = {'max_depth' : max_depth,
           'n_estimators' : n_estimators,
           'sub_sample' : sub_sample,
           'learning_rate' : learning_rate}

    model.set_params(**params)

    return np.mean(-1 * cross_val_score(model, train_x1, train_y,
                            cv = 5, n_jobs = -1, scoring = 'neg_mean_squared_error'))

xgbc_study = optuna.create_study(direction = 'minimize')
xgbc_study.optimize(objective, n_trials = 10)

xgbc_study.best_params
optuna_rfc_mse = xgbc_study.best_value

model.set_params(**xgbc_study.best_params)
model.fit(train_x1, train_y)
xgbc_optuna_pred = model.predict(test_x1)
xgbc_optuna_mse1 = mean_squared_error(test_y, xgbc_optuna_pred)

The full warning is:

Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.

I want MSE as my metric of choice.

mkrieger1
  • 19,194
  • 5
  • 54
  • 65
spectre
  • 717
  • 7
  • 21

1 Answers1

6

Just as described here, try to add eval_metric to your .fit:

model.fit(train_x1, train_y, eval_metric='rmse')

as optimizing rmse and mse is leading towards the same results.

Rafa
  • 564
  • 4
  • 12
  • I made the changes `model.fit(train_x1, train_y, eval_metric='mse')` but is gives me error as `XGBoostError: [21:13:39] ..\src\metric\metric.cc:49: Unknown metric function mse` – spectre Aug 04 '21 at 15:45
  • 1
    because there is no `mse` metric. that's why I suggested `rmse`, as it's basically the same (one is a root of another one) – Rafa Aug 05 '21 at 07:25
  • you can define `mse` yourself if you want, but why would you, when you have `rmse` and it's only for optimization purposes (as you create `xgbc_optuna_mse1` later anyway? – Rafa Aug 05 '21 at 07:26
  • Ok got it! Thanks! – spectre Aug 05 '21 at 09:41