0

I am trying to optimize a classification model for a certain performance metric using hyperparameter tuning. I did not receive the above error until I tried to tune my model specifically for the precision_score metric; when I just used my model as a normal predictor everything ran very smoothly, even when I added hyperparameter tuning it was fine. Only when I tried to optimize it to a certain metric did it throw this error. This is the relevant code:

scorers = {
    
    'precision_score': make_scorer(precision_score),
    'recall_score': make_scorer(recall_score),
    'accuracy_score': make_scorer(accuracy_score)
}
rf = RandomForestRegressor()
cv=KFold(n_splits=5,random_state=1,shuffle=False)
rf_rsearch = RandomizedSearchCV(estimator = rf, param_distributions = random_hypparams, scoring=scorers,refit='precision_score', return_train_score=True, n_iter = 50, cv = cv , verbose=2, n_jobs = -1)
rf_rsearch.fit(OS_x, OS_y)

In this code:

  • random_hyperparams: is just a grid of randomized hyperparameters to be tested to find the best set.
  • OS_x, OS_y are the x and y training set oversampled using SMC with the respective shapes: (1290, 33) (1290,)

The error seems to be occurring in the final line of the chunk of code displayed.

userj
  • 158
  • 11
  • All these 3 scores are only meaningful for classification problems, while you are using a regression model, for which they are meaningless. – desertnaut Dec 23 '21 at 08:28
  • @desertnaut I am doing a classification problem. – userj Dec 23 '21 at 15:14
  • Then I am afraid you are using the wrong model, which evidently is for *regression* ones; try `RandomForestClassifier` instead. – desertnaut Dec 23 '21 at 15:16
  • @desertnaut Yes, I realised that was a silly mistake Thank you! I changed it to the classifier and changed the make_precision line to scorer = make_scorer(precision_score, greater_is_better=True, average="weighted") but now i get this error: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted']. I have no idea when Im getting this error as I have defined the average – userj Dec 23 '21 at 18:28
  • Please open a new question with the full details – desertnaut Dec 24 '21 at 06:32

1 Answers1

0

The error message indicates that the precision_score you are using does not support scoring the types of outputs/targets you are passing. Does this model make multiclass or continuous predictions?

If this is a multiclass predictor, you should be able to optimize according to precision_score, but you need to define an average for the multiclass targets by passing one of ‘micro’, ‘macro’, ‘samples’, or ‘weighted’ to precision_score's average parameter as the default=’binary (see documentation). The accepted answer to this StackOverflow post should help you do that with make_scorer.

If your model is making continuous predictions you will need to select a scorer that supports continuous output/target evaluation. The metrics under "Regression" in the scikit-learn documentation may help.

user13587369
  • 62
  • 6