4

Let's say we tune an SVM with GridSearch like this:

algorithm = SVM()
parameters = {'kernel': ['rbf', 'sigmoid'], 'C': [0.1, 1, 10]}

grid= GridSearchCV(algorithm, parameters)
grid.fit(X, y)

You then wish to use the best fit parameters/estimator in a cross_val_score. My question is, which model is grid at this point? Is it the best performing one? In other words, can we just do

cross_val_scores = cross_val_score(grid, X=X, y=y)

or should we use 

cross_val_scores = cross_val_score(grid.best_estimator_, X=X, y=y)

When I run both, I find that they do not return the same scores so I am curious what the correct approach is here. (I would assume using the best_estimator_.) That raises another question, though, namely: what does using just grid use as a model then? The first one?

Bram Vanroy
  • 27,032
  • 24
  • 137
  • 239

1 Answers1

3

You don't need cross_val_score after fitting a GridSearchCV. It already has attributes that allow you to access cross validation scores. cv_results_ gives you all. You can index into this with the best_index attribute if you want to see only that specific estimator's results.

cv_results = pd.DataFrame(grid.cv_results_)
cv_results.iloc[grid.best_index_]
mean_fit_time                        0.00046916
std_fit_time                         1.3785e-05
mean_score_time                     0.000251055
std_score_time                      1.19038e-05
param_C                                      10
param_kernel                                rbf
params               {'C': 10, 'kernel': 'rbf'}
split0_test_score                      0.966667
split1_test_score                             1
split2_test_score                      0.966667
split3_test_score                      0.966667
split4_test_score                             1
mean_test_score                            0.98
std_test_score                        0.0163299
rank_test_score                               1
Name: 5, dtype: object

Most of the methods you call on a fitted GridSearchCV use the best model (grid.predict(...) gets you the predictions for the best model, for example). This is not true for the estimator. The difference you see is probably comes from that. cross_val_score fits it again, but this time makes the scoring against grid.estimator but not grid.best_estimator_.

ayhan
  • 70,170
  • 20
  • 182
  • 203
  • 1
    So, to make sure that I get this right: GridSearch will test all parameters on all splits? So let's say there are four parameter combinations, and k = 5, then for each split all four combinations are tested? The mean results of all 5 splits are compared and the best one is returned. So in your example, split0, ... split4 are only the results for the splits of the best model? – Bram Vanroy Dec 29 '19 at 12:03
  • Does that also mean that the hyperparameters can differ across splits? I'm sorry for all these questions but the documentation is quite brief in specifics. – Bram Vanroy Dec 29 '19 at 12:04
  • Thank you for the clarification. On the last point: is it always the case that they will be the same? I am asking since the attribute `best_params_` would make me suspect that for all splits the same hyperparameters are *always* used. – Bram Vanroy Dec 29 '19 at 12:25
  • Okay, I think I understand it now. It is a bit confusing for me because doing 5-fold cross validation will fit in fact five models, so it is possible that for training the best hyperparameters differ between these five models. However, that is apparently not how GridSearchCV works. – Bram Vanroy Dec 29 '19 at 13:43
  • I just went through the source code and it seems that `cross_val_score` just calls the fit method of the estimator. So when passing `grid` in my example, I suppose that that would mean that inside the cross validation (cross_val_score) for each split the gridsearch is going through all parameters *again*? That's indeed not what I want! – Bram Vanroy Dec 29 '19 at 13:48
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/205023/discussion-between-bram-vanroy-and-ayhan). – Bram Vanroy Dec 29 '19 at 14:18