0

It seems pretty standard to use cross validation to determine the best parameters. Of course, this is usually a time-consuming process. Are there any shortcuts? Are there other, faster, forms of exploratory analysis that can provide a hint as to which values will be best?

For example, at my current understanding of machine learning and SVM, I might do something like perform an initial grid search in the range of [10e-5, 10e5] at exponents of 10 for C, and then fine tune from there. But is there a way I could quickly estimate that the best C is somewhere between 10e3 and 10e5, and then perform more specific searches?

This question probably applies to most ML techniques, but I happen to be working with SVM right now.

Nicholas Hassan
  • 949
  • 2
  • 10
  • 27
  • 1
    Instead of grid search, you may want to check out bayesian optimization. E.g.: https://www.mathworks.com/help/stats/bayesian-optimization-case-study.html or https://thuijskens.github.io/2016/12/29/bayesian-optimisation/ – thc Apr 18 '17 at 21:06

1 Answers1

1

Yes, this is an area of active research! There has been a lot of work in different approaches to hyper-parameter tuning besides the standard grid search we all know and (maybe?) love.

The area most similar to what you are describing are various bayesian / gaussian process approaches to the problem. This github repo has an implementation and some informative pictures on how it works https://github.com/fmfn/BayesianOptimization . This approach works by treating the parameter optimization problem as another machine learning problem, where we have features for every hyperparameter, and try to predict the performance of various parameter combinations.

That is a high level description of the process, you can read the linked papers/notebooks in the repo for more details.

Raff.Edward
  • 6,404
  • 24
  • 34