4

dtreeviz has an easy and a rather intuitive way to visualize decision trees. When we train using a XGBoost model, there are usually many trees created. And the prediction of the test data would involve a cumulative addition of values of all trees to derive the test target values. How do we go about visualising a representative tree from those trees?

In my attempt to answer this question, I used sklearn California Housing data and trained with XGBoost. Here is the code:

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
import xgboost as xgb

housing = fetch_california_housing()
X_train, X_valid, y_train, y_valid = train_test_split(housing.data, housing.target, 
                                                   test_size = 0.33, random_state = 11)
dtrain = xgb.DMatrix(data=X_train, label=y_train)
dvalid= xgb.DMatrix(data=X_valid, label=y_valid)

# specify xgboost parameters and train the model
params_reg = {"max_depth":4, "eta":0.3, "objective":"reg:squarederror", "subsample":1}
xgb_model_reg = xgb.train(params=params_reg, dtrain=dtrain, num_boost_round=1000, \
      early_stopping_rounds=50, evals=[(dtrain, "train"),(dvalid, "valid")], verbose_eval=True)

As I used early_stopping_rounds, it stopped at the following iteration:

[0] train-rmse:1.46031  valid-rmse:1.47189
[1] train-rmse:1.14333  valid-rmse:1.15873
[2] train-rmse:0.93840  valid-rmse:0.95947
[3] train-rmse:0.80224  valid-rmse:0.82699
...
[308]   train-rmse:0.28237  valid-rmse:0.47431
[309]   train-rmse:0.28231  valid-rmse:0.47429

xgb_model_reg.best_iteration was 260.

Using this best tree, I plotted a dtreeviz tree as follows:

from dtreeviz import trees
from dtreeviz.models.xgb_decision_tree import ShadowXGBDTree

best_tree = xgb_model_reg.best_iteration
xgb_shadow_reg = ShadowXGBDTree(xgb_model_reg, best_tree, housing.data, housing.target, \
                                housing.feature_names, housing.target_names)
trees.dtreeviz(xgb_shadow_reg)

We get this visual: XGBoost best_iteration tree

If I were to use this ShadowXGBDTree to draw the prediction path through this tree for a validation row, it returns a different value that what the model predicts. For illustration, I randomly chose X_valid[50] and plotted its prediction path, as follows:

# predict
y_pred = xgb_model_reg.predict(dvalid)
# select a sample row and visualize path
X_sample = X_valid[50]
viz = trees.dtreeviz(xgb_shadow_reg,
                    X_valid, 
                    y_valid, 
                    target_name='MedHouseVal', 
                    orientation ='LR',  # left-right orientation
                    feature_names=housing.feature_names,
                    class_names=list(housing.target_names),
                    X=X_sample)            
viz

The predicted target value is 2.13 as shown: Decision Tree prediction path

However, y_valid[50] is 1.741 and even y_pred[50] is 1.5196749, where neither match the value shown in the diagram. I guess this is expected, as I am only using this specific tree for path prediction. How should I select a representative tree, then?

Any thoughts how best to approach this issue? Thank you.

Heelara
  • 861
  • 9
  • 17
  • I did the same question here: [how can I get the final tree model?](https://stackoverflow.com/questions/72311192/how-can-i-get-the-final-tree-model) If you have found an answer please let me know! – Chris Jan 17 '23 at 03:34
  • Any luck with this? Only thing I can think is that the order of features got messed up and swapped somehow. – Kevin Jul 07 '23 at 20:18
  • @Chris/Kevin, I decided to pursue a different path to help with explaining a model prediction. – Heelara Jul 10 '23 at 06:29

1 Answers1

2

After exploring this question for some months, I decided to respond with the direction I have chosen to move forth as others seem to be in a similar situation. My primary aim with this question was to figure out if there is a way to explain a prediction from the XGBoost model. Due to the way XGBoost is theoretically defined, however, it does not look feasible to obtain a single representative decision tree. Instead, I decided to perform SHAP analysis to explain its prediction.

Continuing on with the code given in the question, here is the gist of the code to perform SHAP analysis:

import shap

# Create a tree explainer
xgb_explainer = shap.TreeExplainer(
    xgb_model_reg, X_train, feature_names=list(housing.feature_names)
)
data_dmatrix = xgb.DMatrix(data=X_valid,label=y_valid)
y_pred = xgb_model_reg.predict(data_dmatrix)

shap_explainer_values = xgb_explainer(X_valid, y_pred)

For illustration, if we want to explain why y_pred[50] had 1.5196749, for example, we could generate a waterfall plot with this line:

shap.waterfall_plot(shap_explainer_values[50])

Here is the resulting waterfall plot:

enter image description here

From this plot, features longitude and latitude had the biggest effect on this prediction in opposite directions. The house's latitude positively increased by 2 from the batch base value E[f(x)], however its longitude brought down the value by -2.19. Such a representation readily helps to explain a model prediction.

Heelara
  • 861
  • 9
  • 17