Using f-score in xgb

Question

I'm trying use f-score from scikit-learn as evaluation metric in xgb classifier. Here is my code:

clf = xgb.XGBClassifier(max_depth=8, learning_rate=0.004,
                            n_estimators=100,
                            silent=False,   objective='binary:logistic',
                            nthread=-1, gamma=0,
                            min_child_weight=1, max_delta_step=0, subsample=0.8,
                            colsample_bytree=0.6,
                            base_score=0.5,
                            seed=0, missing=None)
scores = []
predictions = []
for train, test, ans_train, y_test in zip(trains, tests, ans_trains, ans_tests):
        clf.fit(train, ans_train, eval_metric=xgb_f1,
                    eval_set=[(train, ans_train), (test, y_test)],
                    early_stopping_rounds=900)
        y_pred = clf.predict(test)
        predictions.append(y_pred)
        scores.append(f1_score(y_test, y_pred))

def xgb_f1(y, t):
    t = t.get_label()
    return "f1", f1_score(t, y)

But there is an error: Can't handle mix of binary and continuous

Guiem Bosch · Accepted Answer · 2020-03-09T08:52:09.597

6

The problem is that f1_score is trying to compare non-binary vs. binary targets and by default this method does binary averaging. From documentation "average : string, [None, ‘binary’ (default), ‘micro’, ‘macro’, ‘samples’, ‘weighted’]".

Anyways, the error is saying that your prediction is continuous like this [0.001, 0.7889,0.33...] but your target is binary [0,1,0...]. So if you know your threshold I recommend you to preprocess your result before sending it to the f1_score function. Usual value of the threshold would be 0.5.

Tested example of your evaluation function. Does not output error anymore:

def xgb_f1(y, t, threshold=0.5):
    t = t.get_label()
    y_bin = [1. if y_cont > threshold else 0. for y_cont in y] # binarizing your output
    return 'f1',f1_score(t,y_bin)

As suggested by @smci a less_verbose/more_efficient solution could be:

def xgb_f1(y, t, threshold=0.5):
    t = t.get_label()
    y_bin = (y > threshold).astype(int) # works for both type(y) == <class 'numpy.ndarray'> and type(y) == <class 'pandas.core.series.Series'>
    return 'f1',f1_score(t,y_bin)

edited Mar 09 '20 at 08:52

answered Mar 24 '16 at 06:56

Guiem Bosch

2,728
1
21
37

2

Ideally you should parameterize the threshold parameter in the function: `xgb_f1(..., threshold=0.5)`, and default it to 0.5. No magic numbers suddenly appearing inside functions without explanation. – smci Mar 02 '20 at 08:31
By the way, in your list comprehension you don't need floats `0., 1.`, and I don't think you even need the `1 if cond else 0 ...` expression, you can just directly use `int(y_cont > threshold)`. Or if was a pandas Series, `y_cont.gt(threshold).astype(int)` – smci Mar 06 '20 at 12:13
1

Thanks for the input! Honestly, it's been so long since the answer that I don't even remember wether the "verbosity" was intentional or just a product of the rush to come up with an answer. Frankly I don't know how to proceed in such cases. I totally get your point but I have the feeling that the extra verbosity here helps with "readability" making the values explicit and not just a boolean conversion. I know that ultimately it's a subjective perception so I'll wait and see if others support your suggestion by upvoting your comment, and if so I'll change it ;-) Does that sound good to you? – Guiem Bosch Mar 07 '20 at 19:41
1

Sure, no worries. What I do in those situations is show both the verbose code idiom for beginners, and the more advanced/efficient one, with a comment above each saying that. – smci Mar 08 '20 at 08:03

Using f-score in xgb

1 Answers1