How to calculate adjusted R2 score for non-linear models

Question

As mentioned in this post, the adjusted R2 score can be calculated via the following equation, where n is the number of samples, p is the number of parameters of the model.

adj_r2 = 1-(1-R2)*(n-1)/(n-p-1)

According this another post, we can get the number of parameters of our model with model.coef_.

However, for Gradient Boosting (GBM), it seems we cannot obtain the number of parameters in our model:

from sklearn.ensemble import GradientBoostingRegressor
import numpy as np

X = np.random.randn(100,10)
y = np.random.randn(100,1)

model = GradientBoostingRegressor()
model.fit(X,y)

model.coef_

output >>> 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-4650e3f7c16c> in <module>
----> 1 model.coef_

AttributeError: 'GradientBoostingRegressor' object has no attribute 'coef_'

After checking the documentation, it seems GBM consists of different estimators. Is the number of estimators equals to the number of parameters?

Still, I cannot get the number of parameters for each individual estimator

model.estimators_[0][0].coef_


output >>> 
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-27216ebb4944> in <module>
----> 1 model.estimators_[0][0].coef_

AttributeError: 'DecisionTreeRegressor' object has no attribute 'coef_'

How to calculate the adjusted R2 score for GBM?

I am still reading about interpretative models, since you mention that R2 score should only be used to evaluate interpretative models. Can we find any interpretative models in sklearn? I guess linear regression, SVM, Gradient Boosting are not interpretative models. — Raven Cheuk, Mar 23 '20 at 11:28
Linear regression is - that's where the R2 concept was developed and used in the first place. Again, don't confuse the relatively recent discussion of interpetable ML models with what I say about interpretative *statistics* models. What I say is that R2 is practically not used in ML at all... — desertnaut, Mar 23 '20 at 11:34
But in any case, the answer has arguably addressed you underlying *coding* issue, which the question was actually about. I could have stopped there instead of offering the extra advice (which was actually not asked for). — desertnaut, Mar 23 '20 at 11:37
Alright. One last question, other than linear regression, can you name one more interpretative model? Maybe I would understand it more after reading one more example of interpretative model. — Raven Cheuk, Mar 23 '20 at 11:53
Not familiar with any, I'm afraid; I, too, am an ML practitioner, and not a statistician. Stats models that generalize LR (GLMs, GAMs) may be.. — desertnaut, Mar 23 '20 at 12:09

desertnaut · Accepted Answer · 2020-03-21T16:33:14.940

Short answer: don't do it (notice that all the posts you link to are about linear regression).

Long answer:

To start with, your definition that

p is the number of parameters of the model

is not correct. p is the number of explanatory variables used by the model (source).

In agreement to this definition, the post you have linked to actually uses X.shape[1] instead of model.coef_; the latter is suggested in a comment, and it is not correct either (see own comment there).

So, if you insist on computing the r-squared for your GBM model, you can always adjust the code from the linked post (after getting your predictions y_pred), taking also advantage of scikit-learn r2_score:

from sklearn.metrics import r2_score

y_pred = model.predict(X)
r_squared = r2_score(y, y_pred)
adjusted_r_squared = 1 - (1-r_squared)*(len(y)-1)/(len(y)-X.shape[1]-1)

But why you shouldn't do it? Well, quoting from an answer of mine in another question:

the whole R-squared concept comes in fact directly from the world of statistics, where the emphasis is on interpretative models, and it has little use in machine learning contexts, where the emphasis is clearly on predictive models; at least AFAIK, and beyond some very introductory courses, I have never (I mean never...) seen a predictive modeling problem where the R-squared is used for any kind of performance assessment; neither it's an accident that popular machine learning introductions, such as Andrew Ng's Machine Learning at Coursera, do not even bother to mention it.

How to calculate adjusted R2 score for non-linear models

1 Answers1