Validate Accuracy of Test Data

Question

I have fit my model with my training data and tested the accuracy of the model using r squared.

However, I want to test the accuracy of the model with my test data, how to do this?

My predicted value is continuous. Quite new to this so open to suggestions.

LR_swim <- lm(racetime_mins ~ event_month +gender + place +
             clocktime_mins +handicap_mins +
              Wind_Speed_knots+ 
             Air_Temp_Celsius +Water_Temp_Celsius +Wave_Height_m,
               data = SwimmingTrain) 
           family=gaussian(link = "identity")
summary(LR_swim)
rsq(LR_swim) #Returns-  0.9722331

#Predict Race_Time Using Test Data
 pred_LR <- predict(LR_swim, SwimmingTest, type ="response")
#Add predicted Race_Times back into the test dataset.
SwimmingTest$Pred_RaceTime <- pred_LR

Accuracy is a metric while modeling the categorical variables. So if you have a non-categorical variables in your model as a dependent variable while using linear regression, R^2 is the right metric. It is nothing but the explained sum of square over total sum of square ratio. It gives you the fitting performance of your data under the OLS assumptions. — maydin, Jul 26 '19 at 11:03
An intuitive metric, although perhaps not so common outside of environmental science, is the [Nash-Sutcliffe coefficient](https://en.wikipedia.org/wiki/Nash%E2%80%93Sutcliffe_model_efficiency_coefficient). Essentially, it measures how well your model predicts observations and is closely related to R². It's implemented in the [`topmodel`](https://cran.r-project.org/web/packages/topmodel/index.html) package as `NSEff`. You pass your observations and your predictions and the function returns a value in the interval (-∞,1] with 1 indicating perfect predictions. — Dan, Jul 26 '19 at 11:13
But broadly it sounds like you're interested in [cross validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)). — Dan, Jul 26 '19 at 11:22
@maydin thanks for your reply, do you know how to apply this method to my test set in r? ie. how to run rsquared for 'pred_LR' in my code — Kate English, Jul 26 '19 at 11:24
@KateEnglish There is a discussion about your question at [this question](https://stackoverflow.com/questions/25691127/r-squared-on-test-data). — maydin, Jul 26 '19 at 11:36

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

To start with, as already pointed out in the comments, the term accuracy is actually reserved for classification problems. What you are actually referring to is the performance of your model. And truth is, for regression problems (such as yours), there are several such performance measures available.

For good or bad, R^2 is still the standard measure in several implementations; nevertheless, it may be helpful to keep in mind what I have argued elsewhere:

the whole R-squared concept comes in fact directly from the world of statistics, where the emphasis is on interpretative models, and it has little use in machine learning contexts, where the emphasis is clearly on predictive models; at least AFAIK, and beyond some very introductory courses, I have never (I mean never...) seen a predictive modeling problem where the R-squared is used for any kind of performance assessment; neither it's an accident that popular machine learning introductions, such as Andrew Ng's Machine Learning at Coursera, do not even bother to mention it. And, as noted in the Github thread above (emphasis added):

In particular when using a test set, it's a bit unclear to me what the R^2 means.

with which I certainly concur.

There are several other performance measures that are arguably more suitable in a predictive task, such as yours; and most of them can be implemented with a simple line of R code. So, for some dummy data:

preds <- c(1.0, 2.0, 9.5)
actuals <- c(0.9, 2.1, 10.0)

the mean squared error (MSE) is simply

mean((preds-actuals)^2)
# [1] 0.09

while the mean absolute error (MAE), is

mean(abs(preds-actuals))
# [1] 0.2333333

and the root mean squared error (RMSE) is simply the square root of the MSE, i.e.:

sqrt(mean((preds-actuals)^2))
# [1] 0.3

These measures are arguably more useful for assessing the performance on unseen data. The last two have an additional advantage of being in the same scale as your original data (not the case for MSE).

Very good explaination. There is one important thing to clarify, MSE, MAE, and RMSE are the metrics for prediction power rather than the fitting performance. However it was asked the prediction power metrics I guess, that is why she confused the R2 for Accuracy. And that makes this answer the right one. — maydin, Jul 26 '19 at 12:05
@desertnaut many thanks for our informative answer, can the R^2 value be computed the same way, with simple r-scripts, such as those used for MSE, MAE and RMSE? — Kate English, Jul 26 '19 at 13:07
@KateEnglish see [Function to calculate R2 (R-squared) in R](https://stackoverflow.com/questions/40901445/function-to-calculate-r2-r-squared-in-r) — desertnaut, Jul 26 '19 at 13:15

Validate Accuracy of Test Data

1 Answers1