How to pass XGBoost fit parameters when using OneVsRestClassifier?

Question

I want to pass the fit's parameters of xgboost through OneVsRestClassifier's fit method.

clf = OneVsRestClassifier( XGBClassifier(objective='binary:logistic', seed=0))
# Want to pass `eval_set` and 'eval_metric' to xgboost model.
clf.fit(X_train, y_train, estimator__eval_metric='aucpr', estimator__eval_set= eval_set_xgboost)

Error: fit() got an unexpected keyword argument 'estimator__eval_metric'

Can you please help me how can I pass the XGBoost fit parameters using OneVsRestClassifier fit method?

Answer 1

XGBoost by default handles the multi-class classification. Refer to this example for more explanations.

With the current framework, you cannot pass fit_params for OneVsRestClassifier. Refer to this issue for more details.

May be, if you can share your intention for wrapping with OneVsRestClassifier, we can guide you appropriately.

Update:

I don't think wrapping with one Vs rest classifier would reduce the overfitting.

Use the simple XGBoost but fine-tune the hyper-parameters.

• First choice would be reducing the learning rate while increasing the number of iterations of training.

The other best options for reducing the overfitting is briefed here

Answer 2

I have recently running to this issue to pass pos_scale_weight dynamically, and below is how I work around the solution. There is currently no automatic pos_scale_weight available for the XGBClassifier class. And we really want to use sklearn OneVsRestClassifier with xgboost for multi-label or multi-class classification problems.

A common way of using OneVsRestClassifier is as below:

clf = OneVsRestClassifier(XGBClassifier(booster='gbtree', objective='binary:logistic'))
clf.fit(X=X_train, y = y_train)

What it OneVsRestClassifier does is: when you call clf.fit, it actually calls the fit method from XGBClassifier to fit X_train, and each target from y_train to fit the training data. In the example below, the clf.fit method is going to: XGBClassifier.fit(X_train, target1) -> XGBClassifier.fit(X_train, target2) -> XGBClassifier.fit(X_train, target3) -> ensemble all three models. If you set the pos_scale_weight to a certain number then each fit will be using the same scale. If the positive ratio across the all three targets are widely different. It will under-fit the target whose positive rate is way lower than the others.

y_train.head()
| target1| target2 |target3|
|--------|---------|-------|
| 0      | 1       | 0     |
| 1      | 1       | 0     |

In my challenge, each label I am predicting has a totally different pos and neg ratio(range from 0.1% to 10%). Below is a method I created. Assume we have X_train as training features, y_train is a matrix of binary labels for each class. We can work around and create a new class that inherited the fit function and pass a weight_array for each array of y_train. OneVsRestClassifier will pass each y from y_train one by one, therefore the weight_array will be calculated separately. This solution is only for binary classification([0,1]) for multi-label. We want to make sure the neg class's weight is 1, pos class's weight to be the (num of neg)/(num of pos).

class XGBClassifierNew(XGBClassifier):
      """
      the inherited class with same method name will override.
      if you start an XGBClassifierNew instance the fit method you called by default will be XGBClassifierNew.fit(). Check the link below for reference.
      https://stackoverflow.com/questions/12764995/python-overriding-an-inherited-class-method
      """  
      def fit(self, X, y, **kwargs):
          pos_ratio = y.sum()/len(y)
          weight = len(y)/y.sum() - 1
          weight_array = y * (weight-1) + 1
          return super().fit(X=X, y=y, sample_weight = weight_array, **kwargs)

clf = OneVsRestClassifier(XGBClassifierNew())
clf.fit(X=X_train, y=y_train)

And the reason why weight_array is an array is because sample_weight takes weight for each instance rather than entire class like pos_scale_weight.

And this method treats entire class's weight(within each label) equally same.

Answer 3

The answer of Chandler Sekai is helpful. One line needs to be changed however:

weight_array = y * weight + 1
(otherwise you give 0 weight to the negative class...)