0

I have to build a classification model in Python using Gradient Booted Decision Tree and get the model parameters (the value at the node) to implement on hardware. As I understand the final result of a Gradient Boosted Decision Tree is a normal Decision Tree classifier with thresholds to classify the input data.

I have read the following posts:

1-Extracting decision rules from GradientBoostingClassifier

2-how to extract decision rules of GradientBosstingClassifier

As they mentioned,

model.estimators_ contains all the individual classifiers that the model consists of. In the case of a GradientBoostingClassifier, this is a 2D numpy array with shape (n_estimators, n_classes), and each item is a DecisionTreeRegressor.

They showed the way to get the threshold for each decision tree used as estimators in the process of building Gradient Decision Tree classifier. I am not sure if model.estimators contains the final decision tree or not. The scikit-learn documents about ensemble classifier also does not mention it.

Please help me how to extract the final parameter (the value at the node) of Gradient Boosted Decision Tree model from scikit-learn. OR if I am misunderstanding something about the Gradient Boosted DT in scikit-learn, please let me know.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Tạ Đình Lượng
  • 15
  • 8
  • What do you mean by final threshold? – gmds Apr 23 '19 at 07:52
  • I mean the value at the node – Tạ Đình Lượng Apr 23 '19 at 08:00
  • This is not a "threshold" - kindly edit the term in your post to clarify (I was just ready to ask this, too); plus, there is not any "final decision tree" in GBT – desertnaut Apr 23 '19 at 08:01
  • ok i will edit the post – Tạ Đình Lượng Apr 23 '19 at 08:04
  • Why exactly the 1st of the linked threads doesn't answer your question? – desertnaut Apr 23 '19 at 08:06
  • I don't know which tree performs best in the `model.estimators_`. My goal was getting the parameter of the tree which gave the best classification result. – Tạ Đình Lượng Apr 23 '19 at 08:12
  • In my understanding, GBT uses lots of swallow trees (estimators) to create a model that makes a fewer mistake as more trees are added and the model achieved is a decision tree. If there is no final decision tree please give me some resources of what does the model look like? – Tạ Đình Lượng Apr 23 '19 at 08:23
  • See answer; resource request is explicitly off-topic for SO (but simple googling will pay off). Plus, as you now post it, question is no more about *programming* or `scikit-learn`, but about the theory behind boosted trees; hence, should you have any further queries on the subject, I kindly suggest you address them to [Cross Validated](https://stats.stackexchange.com/help/on-topic) (which already hosts some possibly useful threads on this). – desertnaut Apr 23 '19 at 08:47
  • 1
    Thanks for your clarifications. I will close the topic here. – Tạ Đình Lượng Apr 24 '19 at 01:19

1 Answers1

0

I am not sure if model.estimators contains the final decision tree or not [...] OR if I am misunderstanding something about the Gradient Boosted DT

It seems that you do misunderstand a crucial detail: in GBT there is not any "final" decision tree; the way GBT works is roughly:

  • Each tree in the ensemble performs the classification according to its own threshold
  • The outputs of all the trees in the ensemble are weighted-averaged, in order to produce the ensemble output

From your comments:

My goal was getting the parameter of the tree which gave the best classification result

Again, this has nothing to do with boosting, which, as you correctly point out in your next comment, grows trees sequentially, with each tree focusing on the "mistakes" of the previous ones; but

and the model achieved is a decision tree

is not correct, as I have already explained (the final model is the whole additive ensemble). Hence, selecting any single tree does not make any sense here.

Given these clarifications, the 1st of the threads you have linked to gives exactly how to extract the rules (thresholds) for all the trees in the ensemble (which, to be honest, don't know if it is really useful in practice).

desertnaut
  • 57,590
  • 26
  • 140
  • 166