19

I am trying to use AdaBoostClassifier with a base learner other than DecisionTree. I have tried SVM and KNeighborsClassifier but I get errors. What are the classifiers that can be used with AdaBoostClassifier?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
vdesai
  • 813
  • 1
  • 6
  • 15

3 Answers3

44

Ok, we have a systematic method to find out all the base learners supported by AdaBoostClassifier. Compatible base learner's fit method needs to support sample_weight, which can be obtained by running following code:

import inspect
from sklearn.utils.testing import all_estimators
for name, clf in all_estimators(type_filter='classifier'):
    if 'sample_weight' in inspect.getargspec(clf().fit)[0]:
       print name

This results in following output:

AdaBoostClassifier,
BernoulliNB,
DecisionTreeClassifier,
ExtraTreeClassifier,
ExtraTreesClassifier,
MultinomialNB,
NuSVC,
Perceptron,
RandomForestClassifier,
RidgeClassifierCV,
SGDClassifier,
SVC.

If the classifier doesn't implement predict_proba, you will have to set AdaBoostClassifier parameter algorithm = 'SAMME'.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
vdesai
  • 813
  • 1
  • 6
  • 15
  • LinearSVC: depends on the loss ;) but the nyou would probably use the LogisticRegression class. GMM is no classifier. How did you select the classes to test? There is an iterator in utils.testing for iterating over all classifiers ;) – Andreas Mueller Aug 20 '13 at 18:47
  • Thanks, Andreas. LinearSVC irrespective of the choice for loss doesn't support sample_weight. Yes, GMM is not a classifier :) and I have taken it out. I actually wanted to try all the available classifiers and see which ones can be used as base learner. These were the ones I could identify by looking at the scikit website. – vdesai Aug 21 '13 at 21:52
  • 1
    Huh, you are right about LinearSVC. There are a lot more classifier:``from sklearn.utils.testint import all_estimators; print(all_estimators(type_filter='classifier'))`` – Andreas Mueller Aug 24 '13 at 09:04
  • Thanks, this is a very nice way to list out all the classifiers. The testint in the snippet needs to be testing, i.e., `from sklearn.utils.testing import all_estimators; print(all_estimators(type_filter='classifier'))` – vdesai Aug 25 '13 at 23:14
5

You shouldnot use SVM with Adaboost. Adaboost should use weak-classifier. Using of classifiers like SVM will result in overfitting.

Devashish Thakur
  • 93
  • 2
  • 6
  • could someone please provide a reference about the fact that 'using of classifiers like SVM will result in overfitting.' in adaboodt? – Filippo Neri Jul 16 '21 at 10:43
2

Any classifier that supports passing sample weights should work. SVC is one such classifier. What specific error message (and traceback) do you get? Can you provide a minimalistic reproduction case for this error (e.g. as a http://gist.github.com )?

ogrisel
  • 39,309
  • 12
  • 116
  • 125
  • Yes, you are right it works with SVC. Earlier, I was creating the classifer like this: clf = AdaBoostClassifier(n_estimators=100, base_estimator=SVC()); This doesn't calculate the probability. However, changing it to: clf = AdaBoostClassifier(n_estimators=100, base_estimator=SVC(probability=True)); I get a clean run. – vdesai Aug 19 '13 at 14:07
  • I am still having trouble with KNeighborsClassifier. Snippet of the code and traceback are posted at: https://gist.github.com/vijayvd/6269621 – vdesai Aug 19 '13 at 14:16
  • 2
    The error message is: `TypeError: fit() got an unexpected keyword argument 'sample_weight'`. This is because `KNeighborsClassifier` does not support weighted samples hence cannot be used in conjunction with `AdaBoostClassifier` as explained in its docstring. – ogrisel Aug 19 '13 at 14:47
  • Implementing sample weights for KNeighborsClassifier should be pretty straight forward. I have no idea what a boosted KNN looks like, though. Are you sure that is a good idea? – Andreas Mueller Aug 19 '13 at 20:03
  • I was getting pretty good score using KNN and was thinking of improving it using adaboost. But no particular reason to believe it would perform well. – vdesai Aug 19 '13 at 21:17
  • @AndreasMueller: good question. Some kind of boosting might work for k-NN with high enough k, but the first interpretation of sample weighting I can think of for k-NN would change not the highly weighted points but *their neighbors'* predicted values :) – Fred Foo Aug 20 '13 at 00:20
  • @larsmans I think you are right ^^ That is highly odd but not even the trouble I was thinking about. I just tried to picture how it might look like if it was successful... and gave up. – Andreas Mueller Aug 20 '13 at 18:50